System and Method for Identifying Feature in an Image of a Subject

ABSTRACT

A method and system is disclosed for analyzing image data of a subject. The image data can be collected with an imaging system in a selected manner and/or motion. The image data may include selected overlap and be acquired with an imaging system that generates a plurality of perspectives for more than one location. An automatic system and method may then define or identify various features and/or allow for registration for alternative image data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.63/283,762, filed on Nov. 29, 2021, entitled “Feature Detection of aPlurality of Images.” This application includes subject matter similarto that disclosed in concurrently filed U.S. patent application Ser. No.______, (Attorney Docket No. 5074A-000239-US); Ser. No. ______,(Attorney Docket No. 5074A-000241-US); and Ser. No. ______, (AttorneyDocket No. 5074A-000249-US). The entire disclosures of all of the aboveapplications are incorporated herein by reference.

FIELD

The present disclosure relates to imaging a subject, and particularly toa system to acquire image data for generating a selected view of thesubject and identifying and/or classifying features within the image ofthe subject.

BACKGROUND

This section provides background information related to the presentdisclosure which is not necessarily prior art.

A subject, such as a human patient, may undergo a procedure. Theprocedure may include a surgical procedure to correct or augment ananatomy of the subject. The augmentation of the anatomy can includevarious procedures, such as movement or augmentation of bone, insertionof an implant (i.e., an implantable device), or other appropriateprocedures.

A surgeon can perform the procedure on the subject with images of thesubject that are based on projections of the subject. The images may begenerated with one or more imaging systems such as a magnetic resonanceimaging (MRI) system, a computed tomography (CT) system, a fluoroscopy(e.g., C-Arm imaging systems).

SUMMARY

This section provides a general summary of the disclosure, and is not acomprehensive disclosure of its full scope or all of its features.

According to various embodiments, a system to acquire image data of asubject may be an imaging system that uses x-rays. The subject may be aliving patient (e.g., a human patient). The subject may also be anon-living subject, such as an enclosure, a casing, etc. Generally, theimaging system may acquire image data of an interior of the subject. Theimaging system may include a movable source and/or detector that ismovable relative to the subject.

An imaging system may include a movable source and/or detector to createa plurality of projections of a subject. The plurality of projectionsmay be acquired in a linear path of movement of the source and/ordetector. The plurality of projections may then be combined, such as bystitching together, to generate or form a long view (also referred to asa long film). The long view may be a two-dimensional view of thesubject. In various embodiments, however, the long film may also be athree-dimensional (3D) image. The 3D image may be reconstructed based onimage data acquired with the imaging system.

In various embodiments, the imaging system may acquire a plurality ofprojections at different perspectives relative to the subject. Thedifferent perspectives may be generated due to a parallax effect betweendifferent paths of x-rays from a single source to a detector through thesubject. The parallax effect may allow for different views of the sameposition of the subject. The parallax effect may be formed due to afilter having a plurality of slits or slots through which the x-rayspass and impinge upon the detector. Accordingly, movement of the sourceand/or detector relative to the subject may allow for acquisition of aplurality of projections through the subject including a parallaxeffect. The plurality of projections may then be stitched to form aplurality of long views of the subject due to movement of the sourceand/or detector. An imaging system may include that disclosed in U.S.Pat. No. 10,881,371 to Helm et al., incorporated herein by reference.

In one or more of the projections, a feature may be identified, such asa selected edge or portion. For example, a selected one or morevertebrae may be identified in each of a plurality of projections. Thevertebra may be a specific vertebra, such as L5, T3, etc. Variousprojections that include the same portion may then be combined, such asstitched together. The identification may then be incorporated orapplied to the stitched image.

The identification may be performed in one or more manners, as discussedherein. For example, an edge detection algorithm may be applied todetermine edges and/or identify portions based thereon. One or moremachine learning systems may be used to identify one or more features,such as an edge or a portion. The machine learning system may be used toidentify selected portions in one or more projections and/or a stitchedimage.

Further areas of applicability will become apparent from the descriptionprovided herein. The description and specific examples in this summaryare intended for purposes of illustration only and are not intended tolimit the scope of the present disclosure.

DRAWINGS

The drawings described herein are for illustrative purposes only ofselected embodiments and not all possible implementations, and are notintended to limit the scope of the present disclosure.

FIG. 1 is an environmental view of an imaging system in an operatingtheatre;

FIG. 2 is a detailed schematic view of an imaging system with a sourceand detector configured to move around a subject, according to variousembodiments;

FIG. 3 is a top plan view of a slotted filter body, according to variousembodiments;

FIG. 4A and FIG. 4B are schematic illustrations of a slotted filterassembly relative to a source and a detector;

FIG. 5 is a schematic illustration of acquiring a plurality ofprojections in intermediate images, according to various embodiments;

FIG. 6 is a schematic illustration of a formation of a long view with aweighting function;

FIG. 7 is a schematic view of a plurality of types of image dataacquisition;

FIG. 8 is a flow diagram for a labeling and classification method,according to various embodiments;

FIG. 9 is a flow diagram for a labeling and classification method,according to various embodiments;

FIG. 10 is a flow diagram for a labeling and classification method,according to various embodiments;

FIG. 11 is an exemplary illustration of a labeled and classified image;

FIG. 12 is a flow diagram of a registration process, according tovarious embodiments;

FIG. 13 is a tree diagram of a multi-scale masking process forregistration, according to various embodiments;

FIG. 14 is a graphical illustration of the process of FIG. 13 ; and

FIG. 15 is an exemplary illustration of a result of the registrationprocess of FIG. 12 .

Corresponding reference numerals indicate corresponding parts throughoutthe several views of the drawings.

DETAILED DESCRIPTION

Example embodiments will now be described more fully with reference tothe accompanying drawings.

A subject may be imaged with an imaging system, as discussed furtherherein. The subject may be a living subject, such as a human patient.Image data may be acquired of the human patient and may be combined toprovide an image of the human patient that is greater than any dimensionof any single projection acquired with the imagining system. It isunderstood, however, that image data may be acquired of a non-livingsubject, such an inanimate subject including a housing, casing, interiorof a super structure, or the like. For example, image data may beacquired of an airframe for various purposes, such as diagnosing issuesand/or planning repair work.

Further, the image data may be acquired having a plurality ofprojections that may be generated by dividing a single projection areainto a plurality of projections. As discussed further herein, an imagingsystem may include a filter or construct that divides a beam, such as anx-ray cone beam, into a plurality of portions (e.g., fans). Each of thefans may be used to acquire image data of the subject at a singleposition, but due to the division of a cone into a plurality of distinctportions, such as fans, a single cone projection may include a pluralityof projections due to the fans. In various embodiments, three slots maybe used to generate three fans. The source may also and/or thereaftermove relative to the subject to acquire the plurality of distinctprojections at a plurality of positions relative of the subject to thesource.

With reference to FIG. 1 , a schematic view of a procedure room 20 isillustrated. A user 24, such as a surgeon, can perform a procedure on asubject, such as a patient 28. The subject may be placed on a support,such as a table 32 for a selected portion of the procedure. The table 32may not interfere with image data acquisition with an imaging system 36.In performing the procedure, the user 24 can use the imaging system 36to acquire image data of the patient 28 to allow a selected system togenerate or create images to assist in performing a procedure. Imagesgenerated with the image data may be two-dimensional (2D) images,three-dimensional (3D), or appropriate type of images, such as a model(such as a three-dimensional (3D) image), long views, single projectionsviews, etc. can be generated using the image data and displayed as animage 40 on a display device 44. The display device 44 can be part ofand/or connected to a processor system 48 that includes an input device52, such as a keyboard, and a processor 56, which can include one ormore processors, processor module, and/or microprocessors incorporatedwith the processing system 48 along with selected types ofnon-transitory and/or transitory memory 58. A connection 62 can beprovided between the processor 56 and the display device 44 for datacommunication to allow driving the display device 44 to display orillustrate the image 40. The processor 56 may be any appropriate type ofprocessor such as a general-purpose processor that executes instructionsincluded in a program or an application specific processor such as anapplication specific integrated circuit.

The imaging system 36 can include an O-Arm® imaging system sold byMedtronic Navigation, Inc. having a place of business in Louisville,Colo., USA. The imaging system 36, including the O-Arm® imaging system,or other appropriate imaging systems may be in use during a selectedprocedure, such as the imaging system described in U.S. Patent App.Pubs. 2012/0250822, 2012/0099772, and 2010/0290690, all the aboveincorporated herein by reference. Further, the imaging system mayinclude various features and elements, such as a slotted filter, such asthat disclosed in U.S. Pat. No. 10,881,371 to Helm et al. and U.S. Pat.No. 11,071,507 to Helm et al., all the above incorporated herein byreference.

The imaging system 36, when, for example, including the O-Arm® imagingsystem, may include a mobile cart 60 that includes a controller and/orcontrol system 64. The control system 64 may include a processor and/orprocessor system 66 (similar to the processor 56) and a memory 68 (e.g.,a non-transitory memory). The memory 68 may include various instructionsthat are executed by the processor 66 to control the imaging system 36,including various portions of the imaging system 36.

The imaging system 36 may include further additional portions, such asan imaging gantry 70 in which is positioned a source unit (also referredto as a source assembly) 74 and a detector unit (also referred to as adetector assembly) 78. In various embodiments, the detector 78 aloneand/or together with the source unit may be referred to as an imaginghead of the imaging system 36. The gantry 70 is movably connected to themobile cart 60. The gantry 70 may be O-shaped or toroid shaped, whereinthe gantry 70 is substantially annular and includes walls that form avolume in which the source unit 74 and detector 78 may move. The mobilecart 60 may also be moved. In various embodiments, the gantry 70 and/orthe cart 60 may be moved while image data is acquired, including bothbeing moved simultaneously. Also, the imaging system 36 via the mobilecart 60 can be moved from one operating theater to another (e.g.,another room). The gantry 70 can move relative to the cart 60, asdiscussed further herein. This allows the imaging system 36 to be mobileand movable relative to the subject 28, thus allowing it to be used inmultiple locations and with multiple procedures without requiring acapital expenditure or space dedicated to a fixed imaging system.

The processor 66 may be a general-purpose processor or an applicationspecific application processor. The memory system 68 may be anon-transitory memory such as a spinning disk or solid-statenon-volatile memory. In various embodiments, the memory system mayinclude instructions to be executed by the processor 66 to performfunctions and determine results, as discussed herein.

In various embodiments, the imaging system 36 may include an imagingsystem that acquires images and/or image data by the use of emittingx-rays and detecting x-rays after interactions and/or attenuations ofthe x-rays with or by the subject 28. The x-ray imaging may be animaging modality. It is understood that other imaging modalities arepossible, such as other high energy beams, etc.

Thus, in the imaging system 36 the source unit 74 may be an x-rayemitter that can emit x-rays at and/or through the patient 28 to bedetected by the detector 78. As is understood by one skilled in the art,the x-rays emitted by the source 74 can be emitted in a cone 90 along aselected main vector 94 and detected by the detector 78, as illustratedin FIG. 2 . The source 74 and the detector 78 may also be referred totogether as a source/detector unit 98, especially wherein the source 74is generally diametrically opposed (e.g., 180 degrees (°) apart) fromthe detector 78 within the gantry 70.

The imaging system 36 may move, as a whole or in part, relative to thesubject 28. For example, the source 74 and the detector 78 can movearound the patient 28, e.g., a 360° motion, spiral, portion of a circle,etc. The movement of the source/detector unit 98 within the gantry 70may allow the source 74 to remain generally 180° opposed (such as with afixed inner gantry or rotor or moving system) to the detector 78. Thus,the detector 78 may be referred to as moving around (e.g., in a circleor spiral) the subject 28 and it is understood that the source 74 isremaining opposed thereto, unless disclosed otherwise.

Also, the gantry 70 can move isometrically (also referred as “wag”)relative to the subject 28 generally in the direction of arrow 100around an axis 102, such as through the cart 60, as illustrated in FIG.1 . The gantry 70 can also tilt relative to a long axis 106 of thepatient 28 illustrated by arrows 110. In tilting, a plane of the gantry70 may tilt or form a non-orthogonal angle with the axis 106 of thesubject 28.

The gantry 70 may also move longitudinally in the direction of arrows114 along the line 106 relative to the subject 28 and/or the cart 60.Also, the cart 60 may move to move the gantry 70. Further, the gantry 70can move up and down generally in the direction of arrows 118 relativeto the cart 30 and/or the subject 28, generally transverse to the axis106 and parallel with the axis 102.

The movement of the imaging system 36, in whole or in part is to allowfor positioning of the source/detector unit (SDU) 98 relative to thesubject 28. The imaging device 36 can be precisely controlled to movethe SDU 98 relative to the subject 28 to generate precise image data ofthe subject 28. The imaging device 36 can be connected to the processor56 via a connection 120, which can include a wired or wirelessconnection or physical media transfer from the imaging system 36 to theprocessor 56. Thus, image data collected with the imaging system 36 canbe transferred to the processing system 56 for navigation, display,reconstruction, etc.

The source 74, as discussed herein, may include one or more sources ofx-rays for imaging the subject 28. In various embodiments, the source 74may include a single source that may be powered by more than one powersource to generate and/or emit x-rays at different energycharacteristics. Further, more than one x-ray source may be the source74 that may be powered to emit x-rays with differing energycharacteristics at selected times.

According to various embodiments, the imaging system 36 can be used withan un-navigated or navigated procedure. In a navigated procedure, alocalizer and/or digitizer, including either or both of an opticallocalizer 130 and/or an electromagnetic localizer 138 can be used togenerate a field and/or receive and/or send a signal within a navigationdomain relative to the subject 28. The navigated space or navigationaldomain relative to the subject 28 can be registered to the image 40.Correlation, as understood in the art, is to allow registration of anavigation space defined within the navigational domain and an imagespace defined by the image 40. A patient tracker or dynamic referenceframe 140 can be connected to the subject 28 to allow for a dynamicregistration and maintenance of registration of the subject 28 to theimage 40.

The patient tracking device or dynamic registration device 140 and aninstrument 144 can then be tracked relative to the subject 28 to allowfor a navigated procedure. The instrument 144 can include a trackingdevice, such as an optical tracking device 148 and/or an electromagnetictracking device 152 to allow for tracking of the instrument 144 witheither or both of the optical localizer 130 or the electromagneticlocalizer 138. A navigation/probe interface device 158 may havecommunications (e.g., wired or wireless) with the instrument 144 (e.g.,via a communication line 156), with the electromagnetic localizer 138(e.g., via a communication line 162), and/or the optical localizer 130(e.g., via a communication line 166). The interface 158 can alsocommunicate with the processor 56 with a communication line 168 and maycommunicate information (e.g., signals) regarding the various itemsconnected to the interface 158. It will be understood that any of thecommunication lines can be wired, wireless, physical media transmissionor movement, or any other appropriate communication. Nevertheless, theappropriate communication systems can be provided with the respectivelocalizers to allow for tracking of the instrument 144 relative to thesubject 28 to allow for illustration of a tracked location of theinstrument 144 relative to the image 40 for performing a procedure.

One skilled in the art will understand that the instrument 144 may beany appropriate instrument, such as a ventricular or vascular stent,spinal implant, neurological stent or stimulator, ablation device, orthe like. The instrument 144 can be an interventional instrument or caninclude or be an implantable device. Tracking the instrument 144 allowsfor viewing a location (including x,y,z position and orientation) of theinstrument 144 relative to the subject 28 with use of the registeredimage 40 without direct viewing of the instrument 144 within the subject28.

Further, the imaging system 36, such as the gantry 70, can include anoptical tracking device 174 and/or an electromagnetic tracking device178 to be tracked with the respective optical localizer 130 and/orelectromagnetic localizer 138. Accordingly, the imaging device 36 can betracked relative to the subject 28 as can the instrument 144 to allowfor initial registration, automatic registration, or continuedregistration of the subject 28 relative to the image 40. Registrationand navigated procedures are discussed in the above incorporated U.S.Pat. No. 8,238,631, incorporated herein by reference. Upon registrationand tracking of the instrument 144, an icon 180 may be displayedrelative to, including overlaid on, the image 40. The image 40 may be anappropriate image and may include a long film image, 2D image, 3D image,or any appropriate image as discussed herein.

With continuing reference to FIG. 2 , according to various embodiments,the source 74 can include a single assembly that may include a singlex-ray tube 190 that can be connected to a switch 194 that caninterconnect a first power source 198 via a connection or power line200. As discussed above, x-rays can be emitted from the x-ray tube 190generally in the cone shape 90 towards the detector 78 and generally inthe direction from the x-ray tube 190 as indicated by arrow, beam arrow,beam or vector 94. The switch 194 can switch power on or off to the tube190 to emit x-rays of selected characteristics, as is understood by oneskilled in the art. The vector 94 may be a central vector or ray withinthe cone 90 of x-rays. An x-ray beam may be emitted as the cone 90 orother appropriate geometry. The vector 94 may include a selected line oraxis relevant for further interaction with the beam, such as with afilter member, as discussed further herein.

The subject 28 can be positioned within the x-ray cone 90 to allow foracquiring image data of the subject 28 based upon the emission of x-raysin the direction of vector 94 towards the detector 78. The x-ray tube190 may be used to generate two-dimensional (2D) x-ray projections ofthe subject 28, including selected portions of the subject 28, or anyarea, region or volume of interest, in light of the x-rays impingingupon or being detected on a 2D or flat panel detector, as the detector78. The 2D x-ray projections can be reconstructed, as discussed herein,to generate and/or display three-dimensional (3D) volumetric models ofthe subject 28, selected portion of the subject 28, or any area, regionor volume of interest. As discussed herein, the 2D x-ray projections canbe image data acquired with the imaging system 36, while the 3Dvolumetric models can be generated or model image data.

For reconstructing or forming the 3D volumetric image, appropriatetechniques include Expectation maximization (EM), Ordered Subsets EM(OS-EM), Simultaneous Algebraic Reconstruction Technique (SART) andTotal Variation Minimization (TVM), as generally understood by thoseskilled in the art. Various reconstruction techniques may also andalternatively include machine learning systems and algebraic techniques.The application to perform a 3D volumetric reconstruction based on the2D projections allows for efficient and complete volumetricreconstruction. Generally, an algebraic technique can include aniterative process to perform a reconstruction of the subject 28 fordisplay as the image 40. For example, a pure or theoretical image dataprojection, such as those based on or generated from an atlas orstylized model of a “theoretical” patient, can be iteratively changeduntil the theoretical projection images match the acquired 2D projectionimage data of the subject 28. Then, the stylized model can beappropriately altered as the 3D volumetric reconstruction model of theacquired 2D projection image data of the selected subject 28 and can beused in a surgical intervention, such as navigation, diagnosis, orplanning. The theoretical model can be associated with theoretical imagedata to construct the theoretical model. In this way, the model or theimage data 40 can be built based upon image data acquired of the subject28 with the imaging device 36.

With continuing reference to FIG. 2 , the source 74 may include variouselements or features that may be moved relative to the x-ray tube 190.In various embodiments, for example, a collimator 220 may be positionedrelative to the x-ray tube 190 to assist in forming the cone 90 relativeto the subject 28. The collimator 220 may include various features suchas movable members that may assist in positioning one or more filterswithin the cone 90 of the x-rays prior to reaching the subject 28. Oneor more movement systems 224 may be provided to move all and/or variousportions of the collimator 220. Further, as discussed further herein,various filters may be used to shape the x-ray beam, such as shaping thecone 90, into a selected shape prior to reaching the subject 28. Invarious embodiments, as discussed herein, the x-rays may be formed intoa thin fan or plane to reach and pass through the subject 28 and bedetected by the detector 78.

Accordingly, the source 74 including the collimator 220 may include afilter assembly, such as that disclosed in U.S. Pat. No. 10,881,371 toHelm et al., incorporated herein by reference. The filter assembly mayinclude one or more portions that allow for moving a filter relative tothe x-ray tube 190 to shape and/or position the x-rays prior to reachingthe subject 28. For example, with reference to FIG. 3 , the filterassembly may include a slotted filter 300. The slotted filter 300 may beincluded in the filter assembly that is formed of one or more members.For example, the slotted filter 300 that may be sandwiched between orplaced between one or more members. Nevertheless, for the subjectdiscussion the slotted filter 300 will be discussed, briefly. Asdiscussed herein, the slotted filter 300 may be used to filter and shapethe beam from the x-ray source 74 such that three separate fans arecreated for generating image data of the subject 28.

The slotted filter 300 may include dimensions, as discussed furtherherein. The slotted filter 300 may be formed of a selected material suchas tungsten carbide having a selected amount of tungsten, such as about90% minimum tungsten. In various embodiments, the tungsten carbide isANSI grade C2 tungsten carbide. The slotted filter 300 further includesa selected number of slots or slits that are formed through the slottedfilter 300, such as a first slot 340, a second or middle slot 344, and athird slot 348. The slots 340, 344, 348 may be used to form selectedx-ray beams, volumes, or areas, such as fans, when positioned to limitpassage of the beam in the cone 90. Thus, the slotted filter 300 doesnot allow the entire cone 90 to pass to the subject 28 when positionedin the beam by the collimator 220.

Generally, the slotted filter 300 will block all or substantially all ofthe x-rays, save for the x-rays that pass through the slots 340, 344,348. Accordingly, x-rays that engage the detector 78 when passingthrough the slotted filter 300 are limited to only those x-rays thatpass through the slots 340, 344, 348. It is understood by one skilled inthe art that the filter assembly may include additional portions inaddition to the slotted filter 300 that may assist in refining and/orselecting spectral content of the x-rays that pass through the filterassembly 260.

The slotted filter 300 includes various features including the slots340, 344, 348. The slotted filter 300 includes a main body or member 352through which the slots 340, 344, 348 are formed. The main body 352 mayhave a selected thickness 354 (FIG. 4A) between a first surface 320 anda second surface 330 of the slotted filter 300. The thickness 354 may beabout 0.01 in to about 1 in, including about 0.01 in to about 0.1 in,and further including about 0.07 in to about 0.1 in and further about0.09 in (about 2.2 mm). It is understood that the thickness 354 of themain body 352 may be used to form or define the x-rays that pass throughthe slotted filter 300. The main body 352 may include further dimensionsfor various purposes, however, these dimensions may be based upon thesize of the collimator or other appropriate constrictions. Nevertheless,in various embodiments, the main plate 352 of the slotted filter 300 mayinclude a length dimension 356 between terminal ends 357, 358 of themain plate 352. The length 356 may be about 0.5 in. to about 2 in., andincluding about 1.4 in. (35 mm). A width dimension 360 may be about 0.1in to about 2 in., and further including about 0.9 in. (22 mm). The mainplate 352 of the slotted filter 300 may include various configurations,such as chamfered or angled corners 364 that may form an angle of about45 degrees relative to the ends of the main body 352.

Again, it is understood, that the slotted filter 300 may include variousconfiguration for fitting in a selected imaging system, such as theimaging system 36, and specific shapes of the exterior may be based uponconfigurations of the imaging system 36. The thickness 354, however, maybe selected to ensure minimal or no x-ray radiation passes through thefilter assembly 260 other than through the slots 340, 344, 348. Invarious embodiments, the slots may be filled with a radio transparentmaterial and/or only be thinned areas rather than complete passages.Further, the slots may be formed in different shapes than slots.Regardless, the slotted member 300 member be used to form a plurality ofx-ray beams or regions, as discussed herein.

With reference to FIGS. 4A and 4B, the slotted member 300, according tovarious embodiments, allows for a formation of three x-ray fans or areasof x-rays including a first fan 440, a second fan 444, and a third fan448 due to the respective slots 340, 344, 348. The three fans are formedby the slotted filter 300 filtering x-rays from the source 190 save forthe area of the slots 340, 344, 348. In other words, slotted filter 300filters the x-rays from the source 190 and allows the x-rays to passthrough the slots 340, 344, 348 to form the fans 440, 444, 448.

As discussed further herein, the three fans 440, 444, 448 allow forgeneration of selected image projections due to an imaging area on thedetector 78. Further, due to angles of formation of the slots, the firstand third fans 440, 448 are not substantially distorted due tointeraction of x-rays with the plate member 352. It is furtherunderstood that the numbering of the slots 340, 344, 348 and therespective fans 440, 444, 448 is merely for clarity of the currentdiscussion, and not intended to require any specific order. Further, itis understood, that slotted filter 300 may include a selected number ofslots, such as less than three or more than three; three slots areillustrated and discussed for the current disclosure. It is understood,however, that the three slots 340, 344, 348 allow for the generation ofa long view in an efficient and fast manner, as discussed furtherherein. Including a selected different number of slots may allow for ageneration of a different number of intermediate images as discussedherein, but is not required.

As discussed above, the slotted filter 300 may be used in the imagingsystem 36 to acquire images of the subject 28. Returning reference toFIG. 2 , the SDU 98 may be moved around the subject 28 within the gantry70. It is understood that the SDU 98 may be moved in any appropriatemanner, and that the imaging system 36 is exemplary. For example, theslotted filter 300 may be used with a C-arm imaging system, or anyappropriate imaging system. Nevertheless, in various embodiments, theSDU 98 may be rotated from a first position to a second position, suchas about 90 degrees apart. For example, as illustrated in FIG. 2 , afirst position of the SDU 98 may include the source 74 directing thex-rays along the cone 90 for the detector 78 which may be generally ananterior to posterior (AP) orientation relative to the subject 28. TheSDU 90 may be rotated 90 degrees, such that the source is at a secondsource position 74′ (which may emit a second beam cone 90′) and thedetector may be moved to a different position such as at a seconddetector position 78′, which may be a lateral (LAT) or side-to-side viewof the subject 28. The SDU 98 may be positioned at either or both of thepositions and a line scan of the subject 78 may be formed.

The line scan may include moving the gantry 70, including the SDU 98,along the long axis 106 of the subject 28 which may also be referred toas a Z-axis or Z-direction of the imaging system 36 generally in thedirection of the double headed arrow 114 which may be, in variousembodiments, along the axis 106 of the subject 28, as illustrated inFIG. 1 . The detector 78 may, therefore, be moved in a linear directionsubstantially with movement only in the direction of the double headedarrow 114 along a Z-axis. The acquired image data may be used to form along film or long view of the subject 28 with the image data acquired atone or both of the positions of the detector 78, 78′ as illustrated inFIG. 2 . The use of slotted filter 300 may be used to generate aplurality of views along the Z axis, as discussed further herein.

As illustrated in FIGS. 4A, 4B, and 5 , the slotted filter 300 may beused to form the three fans 440, 444, 448 that reach or haveattenuations that are detected by the detector 78. Each of the fans 440,444, 448 directly or have attenuations that impinge or contact thedetector 78 at a substantially narrow position or area. The detector 78may include a plurality of excitable or detector regions or portions460. The detector regions 460 may also be referred to as pixels and mayrelate to a single picture element (pixel) that is illustrated on thedisplay 44 in the image 40.

The entire cone 90 from the source 74 may have an area that would exciteor impinge upon the entire surface of the detector 78. However, theindividual fans 440, 444, 448 generally impinge upon only a narrow bandor number of the pixels 460. It is understood that the number of pixelsexcited may include an entire width 464 of the detector 78, but limitedto only a selected length 468 of the detector. For example, therespective fans 440, 444, 448 may impinge upon, assuming that no objector subject is within the path of the x-rays (e.g., an air scan), about10 to about 100 pixels. The number of pixels excited in the dimension468 on the detector 78, however, may be augmented or adjusted dependingupon the distance from the detector 78 of slotted filter 300, the widthof the slots (340, 344, 348), or other appropriate considerations.Nevertheless, each of the respective fans 440, 444, 448 will impingeupon the detector 78 at a substantially narrow position and excite alength 468 of pixels that may be along a substantially entire width 464of the detector 78. A width of 398 of one or more of the slots 340-348may allow the length of pixels 468 to be excited (e.g., generate imagedata) limits or eliminates parallax distortion within the image portioncollected with the imaging system using the slotted filter 300, asdiscussed herein. Again, it is understood that any one or more of thefans may excite a selected portion of the detector that is not an entirewidth of the detector. The collected image data, however, may still beused as discussed herein, such as for feature detection and/orregistration.

Further, the detector 78 may be impinged upon by the three fans 440,444, 448 substantially simultaneously from a single position of thesource tube 190 along the Z axis generally in the direction of thedouble headed arrow 114. The detector 78, therefore, may output threedifferent images or image data for three different positions of thex-ray at each single position of the source tube 190. Movement, of thesource tube 190 of the source 74 generally in the direction of thedouble headed arrow 114, however, may create a plurality of three viewsalong the Z axis, as discussed further herein. Each of the fans 440,444, 448 may be separated by a selected distance, which may also be anangular distance 472.

The imaging system 36 may be used to generate images of the subject 28,for various purposes. As discussed above, the images may be generated ofthe subject 28 for performing a procedure on the subject 28, such as aspinal fusion and/or implants relative to or adjunct to a spinal fusion.In various embodiments, therefore, user 24 may evaluate the subject 28by viewing and evaluating images of the subject 28 for determination ofplacement of selected implants, such as pedicle screws. Accordingly, theimaging system 36 may be used to acquire an image of the subject 28. Theimage system 36 may be used to acquire one or a plurality ofprojections. As further discussed above, the detector 78 detects x-raysthat pass through or are attenuated by the subject 28. Generally,however, the detector 78 detects a single projection at a time. Theimaging system 36, including the control system 64, either alone or incombination with the processor system 48, may generate a long film orlong view of the subject 28 by accumulating and combining (e.g.,stitching) a plurality of projections of the subject 28. In variousembodiments, the imaging system 36, therefore, may be operated toacquire a plurality of images.

According to various embodiments, for example, less than the entiresubject 28 may be imaged. The acquisition of image data of the subject28, such as a spine 28 s of the subject 28, may be made by moving theimaging system 36, including the SDU 98, in the selected manner. Forexample, as discussed above, a linear or Z-axis image may be acquired ofthe spine 28 s of the subject 28. The source 74 may be moved with theslotted filter 300 to filter the cone 90 to generate or form the fans440, 444, 448 that impinge on the spine 28 s to generate the variousprojections.

Each of the projections and/or at each of the projection positions, eachof the slots in the slotted filter 300 may allow for the acquisition ofa different “view” of the subject 28 during scanning of the subject 28.For example, each of the three fans 440, 444, 448 acquire a projectionat a single position of the SDU 98. Accordingly, at each view theperspective of the subject 28 may be different. A three-dimensionalmodel of the subject 28 may be reconstructed using the plurality ofviews of the subject 28 acquired even during the line scans of thesubject. A line scan of the subject, as discussed above, may be asubstantially linear movement, such as generally parallel with the longaxis 106 of the subject 28. Thus, the SDU 98 may not rotate around thesubject 28 during the acquisition of the linear scan. Nevertheless, theplurality of projections from the various perspectives, as discussedherein, may be used to reconstruct a three-dimensional model of thesubject 28 using the single or two line scans (e.g. AP and lateral linescans). These plurality of projections from various perspectives mayalso be used to identify and/or localize items or features in the imagedata (e.g., high-contrast objects, such as bony anatomy or implants).The localized position from each of the more than one slot projectionsmay also be used to generate a three-dimensional model of the subjectthat is imaged. The different position in the plane determined in eachof the projections may be used to generate the 3D model, as isunderstood in the art.

In various embodiments, turning reference to FIGS. 5 and 6 areconstruction of a long view 704 may be made as disclosed in U.S. Pat.No. 10,881,371 to Helm et al. and U.S. Pat. No. 11,071,507 to Helm etal., all of the above incorporated herein by reference. Thereconstruction may include various intermediate reconstructions and afinal complete or long reconstruction. The intermediate reconstructionsmay be based on the one or more individual slot projections and thecomplete reconstruction on the individual slot projections and/or theintermediate reconstructions.

The reconstruction of the long view (also referred to herein asreconstructed long view) generally includes various features and stepsthat may be included as instructions, such as with an algorithm, thatare executed by one or more processors or processor systems. Forexample, the imaging system processor 66 and/or the processing system 48having a processor 56, may execute instructions to generate the longview based upon the plurality of acquired projections. As discussedabove, operation of the imaging system 36 may acquire the plurality ofprojections, such as with the slotted filter assembly 260. Accordingly,the imaging system 36 may generate projections that are based uponx-rays detected by the detector 78.

The x-ray projections may be acquired at the detector 78 with each ofthe three slots that generate the respective fans 440, 444, 448. Each ofthe three fans 440, 444, and 448 will generate three separate series ofimages or projections 560, 564, 568, respectively. Each of the series ofprojections includes a plurality of projections that are acquiredsubstantially simultaneously as sets of projections through the slottedfilter 300 when the SDU 98 is at a single position. For example, thefirst series 560 may include a first image slice 560 i that will beacquired at the same position of the SDU 98 as a first image slice 564 iand 568 i respective to each of the fans 440, 444, 448. As the SDU 98moves in the selected direction, such as along the axis 106 in thedirection of the arrow 114, a plurality of projections is acquiredthrough each of the slots 340-348 due to each of the fans 440, 444, 448.Accordingly, three series 560, 564, 568 of projections are acquired dueto movement of the imaging system 36 along a selected line scan. Thus,each of the slot projections may be made of or include a plurality ofrespective slot projection slices, 560 i, 56 ii, 56 iii, etc.; 564 i,564 ii, 564 iii, etc., 568 i, 568 ii, 568 iii, etc.

The series of projections 560, 564, 568 are the projections from each ofthe three slots. As discussed further herein, although each of the slotsand the respective fans 440, 444, 448 are used to generate respectiveseries of projections 560, 564, 568, all of the image projections may beused to generate the long view that is reconstructed. Accordingly, theinput of the x-ray projections from all three slots may include input ofall three series of projections 560, 564, 568 which may be analyzed orevaluated separately, in various portions of the reconstruction, andthen combined to form the final long view, as discussed further herein.Each of the image slices for each of the series (e.g., 560 i, 564 i, and569 i) generally and/or substantially are free of parallax distortiondue at least in part to the width of the slot 398 and the correspondinglength 468 excited on the detector. Thus, the slices may be clearer andhave less error or distortion due to the slice width 398.

The reconstruction may further include an input of a motion profile ofthe imaging system 36. The input of the motion profile of the imagingsystem may include the distance traveled, time of distance traveled,distance between acquisition of projections, and other motioninformation regarding the imaging system 36. The motion profileinformation may be used to determine and evaluate the relative positionsof the projections for reconstruction, as discussed herein.

In a first instance, according to various embodiments, the intermediateprojection 610, 614, and 618 may be made based on the respective slotslice projections. The intermediate projections 610-618 may also bereferred to as slot or intermediate films or images. The intermediatereconstructions may be substantially automatic by executing selectedinstructions with one or more of the processor modules or systems. Theintermediate images may be made at a selected focus plane and may begenerated for each of the series 560, 564, 568, as illustrated in FIG. 5. Accordingly, a first intermediate image 610 may be generated basedupon the first series of projections 560. A second intermediate image614 may be based upon the series of projections 564 and a thirdintermediate image 618 may be based upon the third series of projections568. Each of the intermediate images 610, 614, 618 may be stitchedtogether using generally known techniques such as image blending,registration, and view manipulations. These may include blending variousportions of images that are near matches (e.g., determined to be similarportions) to achieve continuity. Registration includes matching oridentifying identical portions of two or more images. Manipulationsallow for altering different images or portions thereof, as discussedherein.

The plurality of projections, also referred to as image data portions,in each of the series or sets, such as the first series 560, are takenat a selected rate as the SDU 98 moves relative to the subject 28. Asillustrated in FIG. 5 , the subject 28 may include the spine 28 s. Asthe SDU 98 moves, for example, the fan 440 is moved a selected distance,such as 1 centimeter (cm) per projection acquisition. Accordingly, eachof the image projections, such as the image projection 560 i, may be thewidth on the detector of the fan 440 and a second image projection 560ii may be 1 cm from the first image projection 560 i and also the widthof the fan 440 on the detector 78. A selected amount of overlap mayoccur between the two image projections 560 i and 560 ii that allows forstitching together into the intermediate projection or image 610, as isgenerally known in the art. Each of the series of projections 560, 564,568 (which may each include image data portions), therefore, may bestitched together at the respective focus plane to generate theintermediate images 610, 614, 618. As discussed above, the focus planemay be initially set at 0 or arbitrarily set at 0 which is generally theisocenter of the imaging system 36 that acquired the plurality ofprojections 560, 564, 568. The intermediate images are generated basedupon the plurality of projections due to movement of the SDU 98.

Each of the three intermediate images 610, 614, and 618 may then becombined to generate a first or initial long view or long film image704. The generation or merging of the various intermediate images, suchas each of the three intermediate images 610, 614, and 618, may includevarious steps and features. In various embodiments, an initialdeformation of various features may be made when generating each of thethree intermediate images 610, 614, and 618. As noted above, each of thethree intermediate images 610, 614, and 618 may be generated based on aplurality of projections. Thus, each of the three intermediate images610, 614, and 618 may include a similar or same feature (e.g.,vertebrae). The amount of deformation to generate each of the threeintermediate images 610, 614, and 618 may be determined and used infurther merging procedures.

According to various embodiments, a weighting function 710 may be usedto assist in the combining of the intermediate images 610, 614, and 618to generate the long view image 704. The weighting function 710 isgraphically illustrated in FIG. 6 . A first weighting function for thefirst fan 440 w illustrates that pixels or image portions may beweighted more for a selected portion (e.g., the left most portion asillustrated in FIG. 6 ) of the long view due to the position of the fan440. The intermediate or central fan 444 may have the function 444 wthat will weight the pixels for the middle of the long view 704 morefrom the updated image 614 u due to the position of the fan 444.Finally, the fan 448 may have the function 448 w to weight the pixels ofa selected portion (e.g., the right most portion as illustrated in FIG.6 ) due to the position of the fan 448 in the long view 704. It isunderstood that other appropriate stitching functions may be used togenerate the initial long view 704 and that the weighting function 710is merely exemplary. Further, a greater weight may be given to theselected intermediate image 610, 614, and 618 that has the leastdeformation when generating the long view. Further, selecteddeformations, such as geometric deformations, may be made whengenerating the long view.

As also understood by one skilled in the art, with reference FIG. 1-6 ,the subject 28 may be imaged. In various embodiments, for example, aspine 28 s of the subject 28 may be imaged. The acquisition of imagedata of the subject 28, such as the spine 28s, may be made by moving theimaging system 36, including the SDU 98, in a selected manner. Forexample, as discussed above, a linear or Z-axis image may be acquired ofthe spine 28 s. As illustrated in FIG. 6 , the source 74 may be movedwith the slotted filter 300 to filter the cone 90 to generate the fans440, 444, 448 that impinge on the spine 28 s. The attenuated x-ray fromthe source of the SDU 98 may then reach the detector 78 for generationof a plurality of projections. As illustrated in FIG. 5A-6 , each of thefans may project from the single source 74 and be formed due to theslotted filter 300 such that three individual fans for projections ofthe spine 28 s on the detector 78. Each of the individual projectionsmay be used to generate a single slot image projection that may becombined or stitched together, as discussed further herein. Further, asthe SDU 98, including the source 74 and the detector 78, move along anaxis such as the axis 106 of the subject 28 in plurality the slottedprojections formed by the slotted filter 300.

Accordingly, the acquisition of the image data may be made bypositioning the subject 28 relative to the SDU 98. The SDU 98 may thenbe operated to move, such as along the axis 106 of the subject 28,including the spine 28 s, to acquire a plurality of image dataprojections of the subject 28. At a selected time, the variousprojections may be used for image identification, featureidentification, registration or the like. For example, each of the slotsof the filter 300 form or provide a plurality of projection slices forthe respective slots. Returning reference to FIG. 6 , for example, theslot 340 is used to generate the fan 440 and as the SDU 98 moves andprovides or forms the plurality of slices 560, for each be referred toas 560 i, 560 ii. Each of these may be combined into a single slot filmor intermediate image, such as in the first intermediate image 610.Accordingly, each of the other slots that form the other fans 440, 448generate respective series of images 564, 568 that may be combined intorespective slot films or intermediate images such as the secondintermediate image 614 and a third intermediate image 618. It isunderstood, as discussed above, that the slot filter 300 may include anumber different than three slots. Accordingly, the three slots andrelated intermediate images is merely exemplary.

Each of the slot films 610, 614, 618 may acquire a selected portion ofthe spine 28 s, or other selected portion of the subject 28.Accordingly, each of the slot film or intermediate images may becombined to form a long film image 704, as illustrated in FIG. 7 . Theintermediate images 610, 614, 618 may overlap a selected amount, thatmay depend upon the size of the imaging system 36, the position of thesubject 28 relative to the SDU 98, or other considerations.Nevertheless, each of the intermediate films 610, 614, 618 may includeoverlap regions. The amount of overlap may be any selected amount suchas from greater than zero percent to about just less than 100 percent,including about 15 percent to about 75 percent. Accordingly, variousportions of the subject 28, such as the spine 28 s, may occur in morethan one and/or all of the intermediate films 610, 614, 618. Asdiscussed further herein, the appearance of the features in various onesof the different intermediate films may assist in identification of thefeatures. Further, the overlap may allow for generation of the long film704 in an appropriate manner. For example, the algorithm or system 710may weight the amount or each pixel in each intermediate images 610,614, 618 that is used when generating long films 704. Accordingly, eachof the fans may have respective weights 440 w, 444 w, 448 w that maychange depending upon the translational position or position of the longfilm relative to the intermediate slices 610, 614, 618. Thus, the longfilm 704 may be generated with the intermediate films, as discussedabove and disclosed in U.S. Pat. No. 10,881,371, incorporated herein byreference. The long film may be used to identify or have identifiedtherein various features, as discussed further herein. Further, the longfilm may be registered to other acquired image data. Thus, the imagingsystem 36 may be used to acquire image data of the subject 28 at anyappropriate time, such as during an intraoperative procedure or duringan operative procedure and it may be used to identify features and/orregistration to other image data of the subject 28.

Each of the intermediate images, such as the three intermediate images610-618, may be made as projections relative to the subject in variousmanners such as an anterior to posterior (AP) view and/or a lateral view(e.g., from a left side to a right side) of the subject 28. Theacquisition of an AP view may be by positioning the source and detector,as illustrated in FIG. 2 , in solid lines to generate the AP projectionsthrough the subject 28. Lateral projections may be made by moving thesource and detector to the phantom lines, as illustrated in FIG. 2 . Itis understood, however, that a plurality of views, such as more than twomay also be acquired with the subject 28 by moving the source anddetector to other positions relative to the subject 28. The discussionherein regarding an AP view and a lateral view, which together may bereferred to as a multi-view or multiple views, is merely exemplary. Itis understood, however, that a process may be performed with only theseviews.

Exemplary items and/or features of the image data may be acquired,classified, and/or used in selected procedures, such as those discussedfurther herein, based upon the types of image data acquired or usingselected image data acquired. With reference, to FIG. 7 , the varioustypes of image data may include multi-slot or multiple-intermediateimages or data 740. The multi-slot image data may include the variousintermediate images such as the intermediate images 610, 614, 618. Asdiscussed above, each of the multi-slot images 610-618 may be taken at asingle perspective relative to the subjects 28. Accordingly, themulti-slot images may be based on a plurality of the slot imagesacquired through the selected slots of the filter member 300 but all befrom a single viewer perspective, such as an AP view.

In addition, and/or alternatively thereto, a multi-view perspective 750may also be acquired. The multiple view 750 may be include respectivelong films or stitch films from each of two perspectives, such as a longor stitched film from an AP perspective 754 and a long or stitched filmfrom a lateral view 758. The multiple view 750, therefore, may beinclude two views that include stitched films or long film that may bestitched as discussed above, such as illustrated in FIG. 7 .

Further, a combination of the multi-slot and multiple view may be usedto generate a plurality of projections or views in amulti-view-multi-slot (MV-MS) projection 780. The MV-MS 780 may includea plurality of the slot films that are based upon the intermediateimages from a selected view or perspective. Accordingly, threeintermediate images may be from an AP view including image orperspective projections 784 that may include three slot film orprojections from each of the slots or intermediate views such as a first784 a, a second 784 b and third 784 c. Each of the three projections maybe the intermediate views from the perspective slots and to the selectedview, such as an AP view. Similarly, three films may be generated from alateral view including a plurality of lateral view film or intermediateimages 788 each of the plurality may include the intermediate images orrespective intermediate images at the lateral view from each of therespective slots including a first intermediate 788 a, a secondintermediate image 788 b, and a third intermediate image view 788 c. Inthe MV-MS, each of the respective intermediate films that would begenerated from the respective slot images, such as the intermediateimages 610-618, discussed above, may be acquired at each of therespective views including an AP and a lateral view. Therefore, forexample, six projections or perspectives may be acquired in the MV-MSconfiguration.

Generally, according to various embodiments, the process or processes asdiscussed further herein allow for detection and/or classification ofone or more features and image data. As discussed further herein, forexample, image data may be acquired of a spine of a subject andidentification or detection of features therein, such as vertebrae, maybe made and classification of the detected features may be made, such asa specific identification of the specific vertebrae (e.g., firstthoracic, or first lumbar).

As discussed above, image data may be acquired of the subject accordingto various procedures and techniques. The image data may be acquired ofthe subject such as with the imaging system, including the imagingsystem discussed above, to acquire a plurality of projections of thesubject, such as through the slot filter 300. The image data, therefore,may be acquired of the subject at a plurality of perspectives either ata plurality of locations or at a single location including the pluralityof perspectives through the slot filter 300. The multiple projectionsmay be used for various procedures, such as identification and/orclassification of features in the image data and/or registration of theimage data to one or more other images and/or the subject 28. Asdiscussed herein, identification of features in an image may beperformed with the plurality projection in a robust and confidentmanner.

With additional reference to FIG. 8 , according to various embodiments,a multi-slot process or procedure 850, as illustrated, may be used togenerate the multi-slot images 740. Initially, the multi-slot process orprocedure 850 is understood to be carried out partially and/or entirelyby executing instructions with a selected processor module or system. Asdiscussed herein, at least portions of the multi-slot process orprocedure 850 may include machine learning portions that are useful forassisting in identifying features (e.g., vertebra) and labeling the same(e.g., vertebra T1). It is understood, however, that various inputs maybe provided manually (e.g., by a user with a selected input) including astarting portion or region or a label of one or more vertebra. Invarious embodiments, however, the process 850 may be substantially,including entirely, automatic to receive the input data 610, 614, 618and output labeled long film 1000, as discussed herein.

As discussed above, each of the slots 340-348 of the slot filter 300 maybe used to generate a plurality of image slides or projections that maybe formed into separate slot images that are generated from each of theseparate slots (also referred to as slot A, slot B, slot C) andtherefore allow generation of the three slot images 610, 614, or 618.The three-slot images may be generated at any appropriate time, such asduring a procedure including a surgical procedure on the subject. It isunderstood, however, that the image data may be acquired of the subject28 at any appropriate time, such as prior to a procedure to assist inplanning, etc. Nevertheless, the image data may also be saved andrecalled for use in the procedure 850 and/or immediately accessed forthe procedure 850. Nevertheless, the procedure 850 may be used toidentify and label various portions in the image data, as discussedfurther herein.

According to various embodiments in the procedure 850 a featureextraction may occur in a first block step 854. The feature extractionmay be performed on each of the three-slot projection or images andtherefore generate three sets of extracted feature data for each of theseparate slots. The feature extraction may extract any appropriatefeature. As discussed herein, according to various embodiments, thefeature extracted includes at least one and up to all of the vertebraein the slot images 610, 614, and 618. It is understood that featureextraction, according to various embodiments, may include at leastvertebra.

The feature extraction block 854 includes first convolutional layers 860may be generated based upon the first slot image or projection 610, asecond convolutional layers 864 may be generated based upon the secondslot image or projection 614, and a third convolutional layers 868 maybe formed and based on the third slot image or projection 618. Thus, thefeatures may be extracted related to the individual slot image orprojection 610-618 and used further in the procedure 850 to assist inthe identification of portions therein. The extracted feature data isillustrated in blocks 872, as discussed herein.

The feature extraction performed in block 854 may be performed in anyappropriate manner. For example, a neural-network or machine-learningsystem may be used to identify features in the feature extraction ordetection block 854. In various embodiments, a machine-learning processRESNET 50 may be used on each of the image-slot projections to generatethe feature extraction data in the portions that may be formed asconvolutional layers 860, 864, 868 relating to each of the slotprojections 610-618, respectively. It is understood, however, that anyappropriate feature extraction process may be used and RESNET 50 (alsoreferred to as residual) it is merely exemplary for the procedure 850.

Further, the features extracted may be determined according to theprocedure 850, which may be a complex multi-step machine learningprocess and/or may be manually identified or set by the user. In variousembodiments, a combination thereof may also be used such as training theRESNET 50 with a selected number of features and/or identifying orlabeling features in a training data set for training the RESNET 50 thatis applied to the selected data, such as the image data of a selected orcurrent subject.

As illustrated in FIG. 8 , the slot images 610-618 are linked to thesubject and are generated through the slot filter 300. Generally, theslot filter 300 is at a single position and three slot image orprojections are generated through each of the respective slots. Theseprojections from each of the respective slots are then placed togetherinto the single slot image projections 610-618 for each of therespective slots. Thus, slot image 610 may be for all projections fromslot 340, slot image 614 from the slot projections 344, and the image618 from the projections through slot 348. The separate slotsprojections are each of the position that is used to acquire one portionof the slot images. The separate slot projections are generally formedat a known angle relative to one another, such as about zero to about 10degrees apart, including about seven degrees between each of theprojections, as illustrated in FIG. 4A thus creating a distance orangular distance 472 between projections of each slot 340-348 at asingle position of the filter 300. This allows each of the projectionsor slot portions to be at known positions relative to one another.Moreover, due to the positioning of the imaging system, including theslot filter 300 for generation of the plurality of slot film images, theslot films may overlap each other a selected amount. Accordingly, asillustrated in FIG. 8 , the first slot image 610 may overlap a portionof the second slot image 614 and/or the third slot image 618. It isunderstood, however, that the feature extraction may occur in each ofthe separate slot images but may be related to each other due to theoverlap of the generation of collection of the slot images relative tothe subject.

The feature extraction process in block 854, including the image data(e.g., any layers thereof in the machine learning process) may beconcatenated to form an image feature concatenate, also referred to asconcatenated feature maps, in block 872. The image feature concatenatein block 872, as noted above, may include each of the features that areextracted from the slot images 610-618 as the various slot images mayoverlap at least a selected amount (including a known amount). Theconcatenated sets may include one for each of the feature extractionsets and referred to respectively as the concatenated layers 860 c, 864c, and 868 c. Therefore, the features in the respective slot images610-618 may be generated as a concatenated feature map or a singleconcatenated feature maps from the three separate input slot images610-618.

With the concatenated feature maps from block 872, a region proposal,which may include one or more regions, may be made in block 880. Theregion proposals may be related to the image data in the concatenatedfeature maps for identification of selected features or elements in theimage data. The region proposals may be used for a region-basedconvolutional neural network (also referred to as an R-CNN). Thus, theregions identified or selected in the region proposal block 880 may beused for the R-CNN or appropriate machine learning system to identifythe features in the image data, as discussed further herein.

Following and performed on the concatenated image feature map 872 may bea region proposal in the region proposal box 880. In the region proposalsection or module 880 of the procedure 850, a region proposal regressionprocess may occur in block 890 and a region proposal classification maybe performed in block 894. Each of these processes, the region proposalregression 890 and region proposal classification 894, are performed onthe concatenated feature map from block 872. Accordingly, the regressionand classification occur on all three of these slot films 610-618,simultaneously. This may, among other aspects allow for creation of aregion proposal for the projection of the same vertebra on all threeslot films in a joint manner so that proposals across different slotfilms can be associated, as discussed herein.

Moreover, the concatenated feature maps 872 may be more efficientlyoperated on, as padding may be performed or used to ensure a similarnumber of features in each slot image 610-618 as each of the portions asthe slot filter 300 may generate image data beyond the bounds of theslot films generated through the other slots. For example, asillustrated and discussed above, the slot relating to the slot film 618may be padded with image data or pixels from the other slot films toensure that the same vertebrae levels are covered amongst each of theseslot film projections.

A classification may be used to classify the features extracted in thefeature extraction block 854. The classification may be based upontraining classifications and may include, for example, vertebrae,surgical instruments in an image, soft tissue or background features, orother appropriate classifications. In various embodiments, for example,a vertebra may be identified and classified in the image as separatefrom all other background information. In various embodiments surgicalinstruments, such as in implant (e.g., a screw), may also and/oralternatively be classified in the image.

A region proposal network (RPN) regression 890 and a RPN classificationnetwork 894 may be performed to assist in identify or evaluating variousfeatures identified in the respective image data or images. In theregression, understanding that the slot film may be substantiallytwo-dimensional image data, various regressor values may be used toevaluate and/or adjust proposals. The regressors may be used to alignthe region proposals to the vertebra. In various embodiments, theproposals may be rough estimations of the location and size of thevertebra. They may overlap, but the proposal may not locate exactly onthe vertebra. The regressors are used to make small adjustments tobetter fit the proposals bounding box to the vertebra. Each of theoutputs from the RPN regression 890 and the RPN classification 894 maybe used to evaluate various regions in the respective slot films and theRPN classification 894 may be used to identify foreground areasincluding proposals that are likely to contain vertebra. Accordingly, inthe region proposal in block 880, a region of interest (ROI) alignmentmay occur to each of the slot films in respective alignment boxes 900,904, 908.

To assist in the alignment, however, the RPN classification in block 894and the RPN regression in block 890 may be used. The regression, asdiscussed above, may include regressors to identify a position of abounding box within the respective image or image data, a size of thebounding box within the image data, and a distance between projectionsof neighboring slot films.

The regressor data points or values may include five regressors, asdiscussed herein. Two regressors include “Δx” and “Δy” that denotedifferences in coordinates relative to the noted distance of centroidsof an identified object or feature from the ground truth. Two regressors“Δw” and “Δh” denote a width and a height from a ground truth box. Afifth regressor “s” is a distance between projections of neighboringslot films. The regressor values may be used to identify or evaluatingthe various features, such as centroids of individual vertebrae withinthe image data. As discussed above, for example, the slot films 610-618may be of a spine of a subject and the identified features may includevertebrae. Accordingly, bounding boxes in respect of centroids ofvertebrae may be identified and the above identified values may be usedto identify the features or a bounding box of feature within the image.

In various embodiments, a single anchor box in an input image may betransformed into a group of three proposals in each of the slot images610-618. The proposals may be assisted by a given and known distance ofeach of the slot images 610-618 from one another (i.e., based upon theknown distance between the slots in the slot filter 300) and allowed orused to generate three proposals in each of the separate slot images610-618 given the known distance. In other words, the proposals can begenerated from the same anchor box is based on the fact that thedistance between projection on slot film A and B is equal to thedistance between slot film B and C. The distance between proposalswithin the same group may be unknown in the projection images and ispart of the prediction from the network (the fifth regress s).

Once the RPN regression and classification have been formed, the regionsare aligned in the ROI alignment blocks 900-908. The ROI blocks are thenconcatenated into the set for an ROI regression and classificationprocess 930. The ROI aligned regions are concatenated in the ROI boxconcatenate block 920 and may then be classified in block 930 includingwith a region-based convolutional neural network (R-CNN) classificationin block 934 and a R-CNN regression in block 938. Two fully connectedlayers 921, 923 with ReLU activations are used to map the proceedingconcatenated box features 920 to intermediate representation for theR-CNN regression 938 and classification 934 that follow. In variousembodiments, there may be three inputs given the input concatenatedfeature boxes 920, as illustrated in FIG. 8 . Similar regressor termsmay be used to perform the regression in the R-CNN regression block 938and the R-CNN classification 934 may then be performed or may also beperformed, such as substantially simultaneously, to perform aclassification of the features in the image data. The R-CNN process 930may allow for output of classification of the features, such asvertebrae, in the image data.

In addition to the classification in the classification block 930,according to various embodiments, an additional module may assist inidentifying or confirming identification or classification of thefeatures in a confirmation block 950, which may also be referred to as abi-directional long-short term memory (Bi-LSTM) module. The confirmationmodule 950 may be a module to assist in confirming and ensuringappropriate classification of the features, such as the vertebrae in theprocedure 850. As illustrated in FIG. 8 , the final long film may be atwo-dimensional long film 1000, such as the two-dimensional long film704 as illustrated in FIG. 6 . The long film 1000, however, may includethe classification of the features in the image data. For example, eachof the vertebrae may be labeled in the long film 1000 that may beotherwise labeled in each of the slot films 610-618, but through theprocess 850 are labeled in the long film 1000. Therefore, the labeledlong film 1000 may include labels of selected vertebrae such as from asixth cervical vertebrae 1002 to a first sacral vertebra 1004. It isunderstood, however, that any appropriate vertebrae may be classifiedand identified within the image 1000. Moreover, the image may be of anyappropriate portion of the anatomy of a subject in portions therein maybe labeled, such as a training of the process 850 that then is used toclassify a current or test subject image. The long film 1000, therefore,include portions of each of the slot films that may be overlapped and/orstitched together, as discussed above.

To assist in the proper classification of the selected features, theconfirmation block 950 may be used, including the Bi-LSTM process, asdiscussed further herein. The Bi-LSTM module 950 allows for contextualclassification of selected features. For example, in the spine of asubject the label of a specific vertebrae is correct, generally, onlywhen correct relative to adjacent vertebrae. For example, in a spineincluding appropriate adjacent vertebrae a third thoracic vertebrae willonly exist between the second thoracic vertebrae and the fourth thoracicvertebrae. Accordingly, as illustrated in FIG. 8 , the vertebrae T31006, the fourth thoracic vertebrae T4 1008 and the fifth thoracicvertebrae T5 1010 will only occur in that specific order from a superiorposition in the image to an inferior position in the image. As thesuperior and inferior positions in the images are known based upon onthe collection of the image data, including the slot films 610-618, thespecific order of adjacent vertebrae may also be used. Accordingly, thisknown order may be used to assist in confirming and/or determiningclassification of vertebrae in the Bi-LSTM module 950.

Generally, the confirmation module may also be referred to as arecurrent module that may be used following the classification in theclassification module 930. It is understood, according to variousembodiments, that the confirmation of recurrent module 950 is optionaland need not be required for classifying the selected features in theimage data. It is understood, however, that the process 850 may be ableto classify the vertebra even when one is missing or replaced with animplant is appropriately trained.

The long or vertical information regarding the position of the vertebraewithin the image may be used to assist in the confirmation 950.Accordingly, after the classification of features, such as the vertebraeclassifications, the vector information regarding the classification ofthe vertebrae may then be used and fed in to three Bi-LSTM layers 952,954 and 956 followed by final linear layer 958. It is understood,however, that any appropriate number of layers may be used, the threebi-directional layers and the final single linear layer is merelyexemplary. The confirmation module 950 allows for a learning of asequential relationship of other vertebrae within the spine. In otherwords, as discussed above, the sequential limitation regarding theidentification or classification of specific vertebrae may be used toassist in confirming or appropriately classifying vertebrae within theimage.

The recurrent module or confirmation module 950 may allow for a lossfunction “L” to be expressed as Equation 1:

L=λ ₁ L _(cls) ^(RPN)+λ₂ L _(reg) ^(RPN)+λ₃ L _(cls) ^(RCNN)+λ₄ L _(reg)^(RCNN)+λ₅ L _(cls) ^(LSTM)

In Equation 1, a weighted loss regarding classification losses L_(cls)with respect to ground truth labels and regression losses L_(reg) arecomputed using a smooth L loss function with respect to ground truthregressors. The weight factors “λ” are included to balance losses of thedifferent terms. In various embodiments, λ_1=λ_2=λ_3=λ_4=1 and λ_5=0.1,where each is a loss function weighting term related to RPNclassification (λ_1), RPN regression (λ_2), RCNN classification (λ_3),RCNN regression (λ_4), and LSTM classification (λ_5). In variousembodiments, however, the coefficients λ may be removed and all setequal to 1.

As discussed above, the process 850 may include a machine learningprocess including one or more modules that allow for determination ofparticular vertebrae and/or other features or objects in images and mayoutput a single image based upon multiple input images. The output maybe used in a selected procedure, such a spinal surgery performed on thesubject 28. As illustrated in FIG. 1 , the subject 28 may be positionedrelative to the imaging system and/or placed in an operating theater forperforming an operation or procedure thereon. Various procedures mayinclude spinal fusions, disc replacements, vertebrae replacements,spinal rod placements, or other appropriate procedures. Accordingly, theprocedure 850 may allow for identification and classification ofvertebrae within the subject 28 for various purposes. For example, thefinal image 1000 including the selected label, such as the labels of thevertebrae 1002 and 1004 may allow for confirmation of a procedure,selection or identification or planning a procedure, or the like.Therefore, the image 1000 may be displayed for viewing by the user 24 asthe image data 40 on the display device 44. In addition, the image 1000may be acquired prior to a procedure and used for planning or the like.

The process 850 may include one or more convolutional neural networks,as discussed above. These may allow for identification of the variousfeatures in the image and generation of the long image 1000.

In addition, the procedure 850 may include various variations thereof toassist in selected outcomes, such as efficiency of calculation,computation of efficiency or speed, or the like. For example, thefeature extraction block 854 and the region proposal block 880 may beperformed as a single machine learning block 1100. The single proceduremay include all of the inputs of the slot films 610-618 for featureextraction and region proposals therein in a single network ormachine-learning process 1100. The procedure 850, therefore, may includean alternative and/or additional processing step or network step ofcombining the feature extraction and region proposal into a singlenetwork. The feature extraction and region proposal may also include orbe performed with a convolutional neural network, or any appropriatemachine learning procedure. Accordingly, in various embodiments, theprocedure 850 may perform the output or produce the output 1000 with anappropriate input subject image based upon the procedure as noted above.In summary, the procedure 850 includes the feature extraction module854, image feature concatenate 872, the region proposal module 880, boxfeature concatenate 920, and the ROI Regression and Classification 930and/or the optional confirmation 950. In various embodiments, theprocedure 850 may be performed sequentially and/or being combinedtogether (at least in part) in a single module 1100.

Further, the procedure 850 may include a training phase that trains theprocedure 850 of the machine learning process. In various embodiments,for example, a plurality of image data may be used to train the machinelearning procedure 850 to achieve a selected output. In variousembodiments, for example, a training data set may be generated basedupon back projection of CT image data generated of a plurality ofsubjects. In various embodiments, a plurality of image data may be usedto train the machine learning procedure 850 that is generated with thesame imaging system as used for the selected output. After training ofthe procedure 850, a subject or current image data may be input into thetrained network to achieve the selected output in the image data 1000.Accordingly, the machine learning procedure 850 may be trained toachieve the selected outcome, such as classification in the long film1000. It is further understood that each current subject or new subjectdata may also be used as training data for training or improving themachine learning process 850 for future or later subject image data.

Turning reference to FIG. 9 , a procedure 1200 may be used to evaluateinput image data for identifying, classifying, and/or confirmingfeatures in input image data. The procedure 1200 may include certainmodules or portions similar to the procedure 850, as discussed above,and similar features or steps were not to be discussed in great detailhere. The procedure 1200 may also be a machine learning system thatevaluates input image data from multiple views. In this regard, theprocedure 1200 is understood to be partially and/or entirely carried outby executing instructions with a selected processor module or system. Asdiscussed herein, at least portions of the procedure 1200 may includemachine learning portions that are useful for assisting in identifyingfeatures (e.g., vertebra) and labeling the same (e.g., vertebra T1). Itis understood, however, that various inputs may be provided manually(e.g., by a user with a selected input) including a starting portion orregion or a label of one or more vertebra. In various embodiments,however, the process 1200 may be substantially, including entirely,automatic to receive the input data 754, 758 and output labeled longfilm(s) 134, 1344 as discussed herein.

As discussed above, the image data acquired with the imaging system orany appropriate imaging system 30 may be collected at various positionsrelative to the subject 28, including an AP view that may include theinput image or images 754 and a left-to-right, or vice versa, LAT viewthat may include the input image or images 758. The multi-view images750, as discussed above in FIG. 7 , may be acquired of the subject atany appropriate time. The images may be acquired before a procedure,during a procedure, or at the end of a procedure. In variousembodiments, for example, the image data may be acquired of the subjectfor planning a procedure, confirming that a planned procedure has beenperformed, or confirming steps and/or planning for steps intermediateduring a procedure.

The image data may be acquired of the subject 28 including the imagingsystem 30. The AP image 754 may include a plurality of slot images thatare stitched together, as discussed above, but all taken in the APperspective or view of the subject 28. Similarly, the lateral view 758may include a plurality of slot images that are stitched together of thesubject 28 that are all taken in the same lateral direction through thesubject 28. The multi-view images 754, 758 may include a selected lengththat is the same (and/or cropped to be the same) of the subject but maybe of different perspectives or views of the subject. Again, asillustrated in FIG. 2 , an AP view may include an acquisition of theimage data with the detector 78 in first position and a lateral view mayinclude acquisition of image data with the detector 78 in a secondposition 78′. In various embodiments, for example, the AP view 754 andthe lateral view 758 may be about 90 degrees offset from one anotherwith respect to the subject 28. For example, the subject 28 may define along axis 106 and the detector 78 is moved 90 degrees within the gantry70 to acquire the two view images. The images 754, 758 may, however, beacquired the long view long axis 106 such that they are substantiallylong views or longitudinal views of the subject 28.

Thus, the procedure 1200 may include input of the AP view 754 andlateral view 758. It is understood, however, that the multiple views ofthe subject 28 may be any appropriate views and AP and lateral views aremerely exemplary. The procedure 1200 may take as inputs multiple viewsrelative to the subject that are offset relative to one another, such asby 50 degrees, 60 degrees, 120 degrees, or the like. Thus, the multipleviews may allow for multiple views of the same portion of the subject28, but the views need not be exactly or nearly 90 degrees offset fromone another. Nevertheless, the procedure 1200 takes inputs from multipleviews which may include the AP view 754 and the lateral view 758.

Thereafter, a feature extraction occurs in a feature extraction block1210. The feature extraction block 1210 may be similar to the featureextraction block 854 discussed above, save for the distinctionsdiscussed herein. The feature extraction may extract any appropriatefeature. As discussed herein, according to various embodiments, thefeature extracted includes at least one and up to all of the vertebraein the views 754, 758. It is understood that feature extraction,according to various embodiments, may include at least vertebra.

The feature extraction block 1210 may include the RESNET 50 network, asdiscussed above. The feature extraction in block 1210, however, mayshare weights between the input images. Thus, the multiple layers may beinspected to extract features in the input image or image data. Asdiscussed above, for example, features may include vertebrae in theimages acquired of the subject 28.

The feature extraction may occur in each of the images separatelythrough the multiple layers represented by the feature extraction layersor convolutional layers 1214 for the AP input 754 and feature extractionlayers or convolutional layers 1218 for the lateral input 758. Each ofthe image inputs 754, 758 may therefore, in the feature extractionmodule 1210, allow for or have separate features that are extractedtherefrom. The convolutional layers 1214, 1218 may then be concatenatedinto extracted feature data also referred to as feature extraction maps1221, 1223, respectively. Thus, the AP images data 754 may form featureextraction maps 1221 and the LAT images 758 may form feature extractionmaps 1223.

The separate feature extraction for each of the input images may then beused in a region proposal module 1240. In the region proposal module1240, a region proposal network (RPN) classification network 1244 may beperformed and a RPN regression 1248 may also be performed in theperspective modules or blocks 1244, 1248. Due to the respective imagedissimilarities, such as due to the differences due to the perspectiveor position relative to the subject of the acquisition, the RPNclassification and regression may be performed separately on theseparate extracted feature inputs 1214, 1218.

The differing views of the subject 28 generate image data includingimage portions or features that may be very different from one anotherdue to the different perspectives and positions of the imaging devicerelative to the subject 28. The feature extraction in block 1210 and theregion proposal in block 1240, therefore, may include procedures andmodules that are applied to each of the input images separately. Forexample, the RPN classification module or block 1244 may be performed onboth of the feature extracted data 1214 from the AP views 754 and thefeature extracted portions 1218 from the lateral views 758. Thus, theclassification of the features in the respective views 754, 758 may beperformed separately on the different views. Similarly, the RPNregression in block 1248 may be performed separately on the differingviews.

Further, regressors may be defined by eight different regressors thatare again differentiated or separated from the two images including afirst Δx, Δy, Δw, and Δh that relates to the AP view 754 and four of thesame regressors that identify or relate to the lateral view 758. Theregressors have the same definition as discussed above in relation tothe procedure 850. The regressors may be used to align the regionproposals to the vertebra. In various embodiments, the proposals may berough estimations of the location and size of the vertebra. They mayoverlap, but the proposal may not locate exactly on the vertebra. Theregressors are used to make small adjustments to better fit theproposals bounding box to the vertebra. Accordingly, the RPNclassification in block 1244 and the RPN regression in block 1248 may beperformed on the separate input image data at the different viewsincluding the AP view 754 and lateral view 758.

As discussed above, the image system 30 may acquire the image data ofthe subject 28 in a selected time or over a selected period. Further,the slot filter 300 that is used in assisting and generating the imagedata is at a known position relative to the detector 78. Therefore, theimaging system may operate to acquire image data of the subject 28 at aknown longitudinal or vertical coordinate along the axis 106 of thesubject 28. Therefore, each of the proposed regions or region boundingboxes may be at a known longitudinal coordinate and therefore may bepaired in an RPN pairing module or block 1260. The region proposals maybe paired in the RPN module 1260 with a joint objectness score computedas a sum of the objectness scores or the two proposals from the twoinputs, respectively. Therefore, while the RPN regression and RPNclassification may be performed on the input data separately due to thedifference of the input image data, the proposals for regions and theirrespective image data may be paired due to the known longitudinalcoordinate which may also be the coordinate of the image data.

With the RPN pairing in block 1260 a region of interest (ROI) alignmentmay be determined in the respective blocks or modules 1264 and 1268. Thealignment may again occur due to the positioning of the respective of 23c proposal regions due to the known longitudinal position of the imagedata acquired of the subject 28.

The aligned image data from the AP and lateral views 754, 758, afterhaving the proposed regions in the region proposal block 1240, areconcatenated are in block 1280. The image data is concatenated via theknown alignment, as discussed above. The concatenated image data inblock 1280 may be used to perform a classification and regressionanalysis or network of the proposals in a classification block 1300. Theclassification of the regions may be performed similar to theclassification as discussed above in an R-CNN classification in block1310. Similarly, an R-CNN regression may occur in block 1320 of theconcatenated image data from block 1280. Two fully connected layers1301, 1303 with ReLU activations are used to map the proceedingconcatenated box features 1280 to intermediate representation for theR-CNN regression and classification that follow. In various embodiments,there may be two inputs given the input concatenated feature boxes 1280,as illustrated in FIG. 9 . After the classification and regressionprocedure in block 1300, the long films may be outputted as respectivelong film AP views 1340 and lateral views 1344. These long views 1340,1344 may include respective classifications or labels based upon aprocedure as discussed above, and a respective long view, such as alabel of a fourth lumbar 1346 in the AP view 1340 and 1348 in thelateral view 1344.

Again, a confirmation or Bi-LSTM module 1360 may optionally be providedbetween the classification module 1300 and the output of the long views1340, 1344. The Bi-LSTM module may be substantially similar to that asdiscussed above including a selected number of bi-directional layers,such as three bi-directional layers 1364, 1368 and 1372 and a linearlayer 1380. These layers may be interconnected via the Bi-LSTM module ornetwork 1360 to assist in confirming or enforcing a sequence on theidentified or classified features. The Bi-LSTM module 1360, however, mayperform or operate substantially similar to the Bi-LSTM module 1950, asdiscussed above.

Therefore, the multi-view process 1200 may be operated to label andidentify features in image data in multiple views. Again, the multipleviews may include (e.g., generated from) the multiple slot image orprojections, as discussed above. Moreover, the multiple views may beinput into the procedure 1200 to be used together, such as in theconcatenated block 1280 and in the R-CNN classification and regressionto classify features identified in the respective image data. Thus, theoutput image data, including the long films 1340, 1344, may includelabels based upon the input data and the procedure 1200.

As discussed above, image analysis may be performed according to variousnetworks on selected image data. The multi-slot analysis may beperformed to identify or label features in the image data and amulti-view may also be used to label features in the image data, asdiscussed above and according to various embodiments. In addition,thereto, a combination may be performed on both a multi-view and amulti-slot in a multi-view-multi-slot (MV-MS) process 1400 to allow foridentification in both a multi-view and a multi-slot image data. Asdiscussed above and illustrated in FIG. 7 , image data may be acquiredfrom each of the slots and formed into the slot films taken along eachof the perspectives, as an AP and a lateral view.

With reference to FIG. 10 , the MV-MS process or network 1400 mayidentify and/or classify or label features in the image data, asdiscussed further herein. Initially, the process 1400 is understood tobe carried out partially and/or entirely by executing instructions witha selected processor module or system. As discussed herein, at leastportions of the process 1400 may include machine learning portions thatare useful for assisting in identifying features (e.g., vertebra) andlabeling the same (e.g., vertebra T1). It is understood, however, thatvarious inputs may be provided manually (e.g., by a user with a selectedinput) including a starting portion or region or a label of one or morevertebra. In various embodiments, however, the process 1400 may besubstantially, including entirely, automatic to receive the input data784, 788 and output labeled long films 1580, as discussed herein.

The input into the process 1400 can include each of the slot films takenfrom each of the respective slots of the slot filter 300, as discussedabove, from multiple views. As illustrated in FIG. 10 , three slots mayuse to generate three-slot films from each of the views including the APview 784 and the lateral view 788 to generate the respective slot films784 a, b, and c and 788 a, b, and c. The image data may be input toallow for feature extraction in each of the slot films from each view infeature extraction block 1420. As discussed above, the featureextraction may occur in an appropriate manner, such as using the RESNET50 network system. The feature extraction in block 1420 may allow forextraction of features in each of the respective slot films 784 a-c and788 a-c. Each of the respective slot films may have a respectiveconvolutional layers 1420 a, 1420 b, 1420 c, 1420 d, 1420 e, 1420 f.Thus, the feature extraction may occur in each of the individual slotfilms and for each of the respective views acquired of the subject 28.

Following the feature extraction in each of the respective slot views, aregion proposal block 1460 occurs. In the region proposal block 1460, aregion proposal may be made in concatenated feature maps based on theviews, including a first concatenated feature map also referred to as afeature extraction maps 1464 for the AP view and a second concatenatedfeature map 1468 for the lateral view. Each of the concatenated featuremaps 1464, 1468 include three feature maps that relate to the same viewfor each of the respective slot films of the respective vies 784, 788.The region proposal 1460 may include a region proposal networkregression 1472 and a region proposal network classification 1476. Theregion proposal regression 1472 and the classification in block 1476 maybe formed similar to that discussed above with the multi-view process1200.

Accordingly, after the regression and classification 1472, 1476, aregion proposal pairing may occur in block 1480, also similar to theprocess 1260 as discussed above. Thus, a total of six proposals forregions of interest may be generated for each of the slot views from theoriginal input and paired in the process 1480. In various embodiments,the pairing in blocks 1480 and 1260 are essentially the same.Longitudinal coordinates of anchor boxes are used for pairing. Thedifference is that in the process 1260 one proposal box is generatedfrom a given anchor box. In the process 1480 three proposals aregenerated from one anchor box, as described above.

Following the region proposal pairing in block 1480 and the regionproposal block 1460, a region of interest regression and classificationblock 1500 may also be performed. The region of interest regression andclassification block 1500 may be similar to the regression andclassification block as discussed above such as the regression andclassification block 1300 in the process 1200. In the regression andclassification block 1500, the six proposals are concatenated into a boxfeature concatenate 1520. The box feature concatenate 1520 may besimilar to the box feature concatenate 1280, as discussed above.

The box feature concatenate 1520 may, therefore, be performed in anetwork or classified in a network also similar to that discussed above.For example, the box feature concatenate 1520 may be placed in a networkincluding an R-CNN regression 1540 and an R-CNN classification 1560. Twofully connected layers 1521, 1523 with ReLU activations are used to mapthe proceeding concatenated box features 1520 to intermediaterepresentation for the R-CNN regression 1540 and classification 1560that follow. In various embodiments, there may be six inputs given theinput concatenated feature boxes 1520, as illustrated in FIG. 10 . Theregression and classification may be similar to that discussed above, aswell. The regression factors may, however, include a Δx, Δy, Δw, Δh, s,Δx′, Δy′, Δw′, Δh′, s′. Each of these regression factors relates to therespective views similar to that discussed above to the multi-viewnetwork. Further, the s, s′ regressors may also be used given themultiple slot films of the MV-MS process 1400. In this manner, theregressors may be used for confirming the classification of each of thefeatures identified in the region proposals. The regressors may be usedto align the region proposals to the vertebra. In various embodiments,the proposals may be rough estimations of the location and size of thevertebra. They may overlap, but the proposal may not locate exactly onthe vertebra. The regressors are used to make small adjustments tobetter fit the proposals bounding box to the vertebra. Therefore, theR-CNN may be applied to the concatenated box features 1520 to inputclassification in each of the views 780 so that views 1580 may includelabels as discussed above. Accordingly, each of the views may include arespective label based upon the analysis of the process 1400.

It is understood that the various views may be combined using variouscombination techniques, such as morphing or stitching. Thus, the inputimage data may be used to identify features and labeled the same inoutput images 1580. As illustrated in FIG. 10 , the label films mayinclude one or more similar to the films discussed above. For example,the output may include a long film 1000 similar to the film 1000discussed and illustrated in FIG. 8 . The labeled film may, however besimilar to the AP film 1340. Further, the output 1580 may include thelong film of the LAT view 1344 similar to the output 1344 illustratedand described in FIG. 9 .

Further, a confirmation block 1600 may be added including the Bi-LSTMprocedure as discussed above. As discussed above, this may include athree bi-directional networks 1610, 1620 and 1630 and a single linearnetwork 1640 for confirmation and/or applying a rigid or a predeterminedorder to the labels in the images. The confirmation or Bi-LSTM block1600 may be used to assist in ensuring a proper or confirmation label ofthe features in the image data.

Accordingly, according to various embodiments, the input image data maybe analyzed according to various procedures, such as a machine-learningprocess that may be used to label and identify images and input imagedata. The input image data may be acquired with selected imaging systemsuch as the imaging system 30. The image data may be analyzed using thetrained machine-learning process, according to the various procedures asdiscussed above. The various procedures may be used according to varioustypes of input data, including that discussed above. For example, theslot films may be acquired individually and analyzed according to themachine-learning process 850. Additionally, and/or alternatively,multiple view image data may be analyzed according to the process 1200.Further, various combinations may be used and analyzed, such asaccording to the machine-learning process 1400. The various processesmay include various steps and analysis, as discussed above, that may beperformed by selected processor modules including those discussed aboveand as generally understood by those skilled in the art. Nevertheless,the output may include image data that may be displayed as images foruse by the user to view labeled features in the image data. The labeledfeatures may be used to assist in performing a procedure and/or aconfirming a planned procedure as also discussed above.

Turning reference to FIG. 11 , the labeled images, according to variousembodiments as discussed above, may be displayed on the display device.Accordingly, the image data may be labeled image data for use by theuser. For example, the display device, which may be in the appropriatedisplay device such as a LCD display, LED display, CRT display, or thelike. Nevertheless, the image may be in the labeled image, such as thatdiscussed above. Thus, the image may include labels of one or morevertebrae in the image data that is displayed as the image. The imagesor image may include the labels that are determined according to thevarious embodiments, as discussed herein. In various embodiments, forexample, the display device may display the image 40 that labelsvertebrae when no surgical instruments are in place, such as the image40 a. The labels may identify or label centroids that have beenidentified in the image data and displayed with the display device 44 a.In addition, and/or alternatively thereto, an image 40 b may bedisplayed that labels vertebra even when a surgical instrument or otheritem is present in the image, such as a screw 1600. The screw 1600 maybe any appropriate screw and is exemplary of an item in the image thatmay be present in addition to anatomical features in the image.Nevertheless, the labeled and displayed image may include features inaddition to anatomical features of the subject 28. Thus, the user 24 mayview the images with the display device 44 b to assist in performingand/or confirming a procedure. The labeled portions of the image may belabeled with or without non-anatomical features, such as surgicalinstruments including implants.

The imaging system 30, or any appropriate imaging system, may be used toacquire image data of the subject 28. The image data may be analyzed, asdiscussed above, including labeling various features in the image data.The features may include anatomical portions in the image data, implantsor surgical instruments in the image data or any other appropriateportion in the image data. According to various embodiments, variousmachine-learning systems, such as networks, may be trained to identifyone or more features in the image data. As discussed above, the imagedata labels or identification may include centroids of vertebra. It isunderstood, however, that various portions of the image data may also beclassified to be identified in the image data. Accordingly, during aselected procedure or at an appropriate time, image data may be acquiredof the subject 28 with an appropriate imaging system, such as theimaging system 30, and features therein may be identified and/orlabeled.

In various embodiments, a procedure may occur on the subject 28, such asplacement of implants therein. Pre-acquired image data may be acquiredof the subject, such as three-dimensional image data including aComputed Tomography (CT), Magnetic Resonance Imaging (MRI), or the like.The image data may be acquired prior to performing any portion of aprocedure on the subject, such as for planning a procedure on thesubject. The pre-acquired image data may be then used during a procedureto assist in performing the procedure such as navigating an instrumentrelative to the subject (e.g., a screw) and/or confirming a pre-plannedprocedure. In various embodiments, image data acquired of the subjectduring a procedure or after the acquisition of the initial or prioracquired image data may be registered to the prior or pre-acquired imagedata. For example, image data may be acquired with the imaging system 30and may be registered to the pre-acquired image data according tovarious embodiments, including those discussed herein.

The registered image data may assist in allowing a user, such as theuser surgeon 24, to understand a position of the subject at a givenperiod of time after the acquisition of the initial pre-acquired imagedata. For example, the subject 28 may have moved and/or be repositionedfor a procedure. Thus, image data acquired with the imaging system 30may be registered to the pre-acquired image data.

The registration to the pre-acquired image data may include variousportions as discussed further herein. Moreover, the registration of theimage data to the pre-acquired image data may include registration of alarge portion of the subject 28. For example, the imaging system 30 mayacquire image data of the subject including several vertebrae, such asfive or more, 10 or more, including about 10, 11, 12, 13, 14, or morevertebrae. As understood by one skilled in the art, the vertebrae maynot be rigidly connected to one another and, therefore, may moverelative to one another over time, such as between acquisition ofpre-acquired data and acquisition of a current image data. Therefore, aregistration process may and/or need to account for the possiblemovement. In various embodiments, therefore, a computer implementedsystem may be operated to account for and/or be flexible enough toaccount for movement of portions in the image data (e.g., vertebrae)relative to one another while being able to determine a registrationbetween the prior acquired image data and the current image data.

As discussed above, and illustrated in various figures including FIG. 5, FIG. 6 , and FIG. 11 , a long film or long view image of the subject28 may be generated with the system 20, including the imaging system 30and/or various processing systems to stitch together various slot filmsand/or slot projections or the subject 28. Therefore, the long film mayinclude a plurality of vertebrae of the subject 28 and variousanatomical features included in the subject 28 including features of thevertebrae, other hard tissues (e.g., ribs, pelvis) and various softtissues, such as cartilage, musculature, etc. The images or projectionsmay be stitched or placed together, as discussed above. In variousembodiments, the reconstruction from the three slots may includeTomosynthesis. This may allow for an image to be generated that is up toabout 64 centimeters in length. The length may relate to a physicallength of the film and/or a physical length of the object, such as thesubject 28, being image that is included in the image data image in thelong film.

The long film, or any appropriate projection image, including those asdiscussed above, may be registered to pre-acquired image data. Thepre-acquired image data may include appropriate image data such athree-dimensional (3D) image data. That may be generated or acquiredfrom various imaging modalities such as CT, MRI or the like. In variousregistration techniques, computer implemented algorithms and/ormachine-learning processes may be used to perform the registration. Forexample, in various embodiments, a patient registration between thethree-dimensional image and the intraoperative or intra-procedure orlater acquired images, which may be two-dimensional images. A deviceregistration may also be performed using known component registrationmethods. Various known component registration methods include thosedisclosed in U.S. Pat. No. 11,138,768, incorporated herein by areference.

With reference to FIG. 12 , a registration procedure system 1700 isillustrated. The registration procedure 1700 may include two mainportions that may be performed sequentially and/or separately. Theregistration method 1700 may include a patient registration 1710 and adevice registration 1720. The patient registration 1710 may generallyregister the pre-acquired image data to a current or intraoperativeimage data of the subject, such as the subject 28. Therefore, thepatient registration or subject registration 1710 may includeregistering image data of the subject 28 that is acquired at twodifferent times. The second registration 1720 may be a device orinstrument registration which may register a tract position of theinstrument or an image position of the instrument to a determinedposition. In the device registration 1720, information regarding theinstrument may be known and viewed, such as known components (e.g., atwo-dimensional model, three-dimensional model, material selection orinclusion) used to assist in the registering or analyzing the image ofthe subject 28 including the instrument or device. Thus, theregistration 1700 may include the two main registration steps or portionincluding the subject registration 1710 and the device registration1720.

The registration 1700, including the two main registration steps orportion including the subject registration 1710 and the deviceregistration 1720 is understood to be carried out partially and/orentirely by executing instructions with a selected processor module orsystem. As discussed herein, at least portions of the registrationprocess may include machine learning portions that are useful forassisting in identifying features (e.g., vertebra) and/or masking thesame. It is understood, however, that various inputs may be providedmanually (e.g., by a user with a selected input) including a startingportion or region or a label of one or more vertebra. In variousembodiments, however, the registration 1700 may be substantially,including entirely, automatic to receive input data, such aspreoperative and current image data and output a registrationtherebetween.

With continuing reference to FIG. 12 , the subject registration 1710performs a registration (also referred to as morphing or non-rigiddeformation) of prior acquired or preoperative image data 1740. Thepreoperative image data 1740 may be acquired at any time prior to acurrent image data or intraoperative image data. Moreover, thepreoperative image data may be any appropriate type of image dataincluding 2-dimension and/or 3-dimensional image data. In variousembodiments, for example, the preoperative image data 1740 may includeCT image data. The CT image data may be generated as a 3-dimensionalimage data of the subject 28. It is understood, however, that anyappropriate image data may be acquired of the subject and preoperativeCT image data is merely exemplary. Other types of image data include MRIimage data, ultrasound image data, or the like. The preoperative imagedata 1740 is acquired prior to a current image data 1744, that is thecurrent image data 1744 may be acquired of the subject 28 at anyappropriate time, such as during an operative procedure, following aportion of an operative procedure or the like. The current image data1744 is acquired of the subject and generally includes at least aportion of the subject that is included in the preoperative image data1740. Thus, the current image data 1744 may include the image data, suchas that discussed above. For example, the current image data 1744 mayinclude image data that is labeled of the subject 28, such asidentifying centroids of vertebral bodies in the image data. The labeledportions of the image may be labeled based upon the processes, asdiscussed above. Thus, the current image data may include identificationof various portions within the image data such as the vertebrae,implants in the image, or other appropriate features. According tovarious embodiments, labels may be applied to portions of the image dataand identification of vertebrae and/or centroids of vertebrae is merelyexemplary.

The subject registration 1710 allows for a registration of thepreoperative image data 1740 to current image data 1744 even if therehas been a deformation or a change in relative position of variouselements with the image data between the preoperative image data 1740and the current image data 1744. For example, as discussed above, thepreoperative image data 1740 and the current image data 1744 may includea plurality of vertebrae. The plurality of vertebrae may be the samevertebrae between the two image data sets 1740, 1744 but may be indifferent relative positions due to movement of the respective vertebraeduring a time period between the acquisition of the preoperative imagedata 1740 and the current image data 1744. Nevertheless, a masking andoptimization subroutine 1750 is operable to allow for registrationbetween the preoperative image data 1740 and the current image data1744. The current image data may also include or be referred to asintraoperative image data, as discussed above. The masking subroutine1750 may include a machine-learning process to allow for training of amachine-learning process to then register the specific or patientspecific preoperative image data 1740 to the current image data 1744.

The registration process 1750 includes the input of the current images1744 that may include multi-view images, as discussed above. Themulti-view images may include an AP slot image or film 1744 a and alateral slot image or film 1744 b. Thus, the current image data 1744 mayinclude a plurality of views such as an AP and a lateral view asdiscussed above. Moreover, as also discussed above, these views may belabeled according to the processes, such as the labeling process MV-MS1400. Similarly, the preoperative image data 1740 may also be labeled,such as the labeling of vertebral centroid 1742. The labeling of thepreoperative image data may be performed in any appropriate manner suchas a manual process (e.g., user identified in the image), an automaticprocess (e.g., the processes disclosed above), or a combination thereof.In various embodiments, a machine-learning process may be used toidentify and label the centroids or portions of the image in thepreoperative image 1740. In various embodiments, a user, such as asurgeon, may alternatively or also identify the centroids or anatomicalfeature or other features in the preoperative image data and may beinput as labels which may include the vertebral centroids 1742.Accordingly, the preoperative image data 1740 and the current image data1744 may be input into the registration subprocess 1750.

In the registration subprocess, a further multi-scale mask subprocess1760 may occur. As discussed herein, the multi-scale masking 1760 mayallow for successively smaller portions of the input image data to bemasked and registered to the current image data. The multi-scale maskingallows for registration when there is deformation or relative change offeatures that are included in both the preoperative image data 1740 andthe current image data 1744. For example, the various vertebrae, such asT4 and T5, may move relative to each other and be in different relativeposition between the preoperative image data 1740 and the current imagedata 1744. Thus, the multi-scale masking subroutine 1760, as discussedfurther herein, may be used to assist in the registration. In variousembodiments, masking process 1760 may require only requires knowledge ofthe vertebral centroids as opposed to a pixel-wise segmentation. Thus,masking may also be referred to as a “local region of support”.

The preoperative image data may then be used to generate synthetic slotimages that may relate to the current image data including a syntheticAP slot image 1770 and a synthetic lateral slot image 1774. Thesynthetic images may be generated such as by forming projections throughthe input preoperative image data 1740 to generate the synthetic images1770, 1774. The projection is generally computed by forward projectionof the preoperative image 1740 through the image data at selectedorientations to generate the synthetic slot images 1770, 1774.

The respective slot images may then be matched or registered to thecurrent image data 1744 in an optimization subroutine 1780. Theoptimization subroutine may generally include an optimization of agradient orientation (GO) metric that is optimization using a covariantmatrix adaptation evolution strategy. Such strategies may include thosedisclosed by Hansen, N. and Ostermeier, A., “Completely derandomizedself-adaptation in evolution strategies.,” Evol. Comput. 9(2), 159-195(2001).

The optimization procedure 1780 optimizes similarity between thesynthetic slot images 1770, 1774 and the current image data 1744, thatcan include equivalent current slot data 1744 a, 1744 b. Theoptimization optimizes the similarity between synthetic slot images1770, 1774 to the current image data 1744 to determine a registration ofthe preoperative image data 1740 (from which the synthetic slot imagesare generated 1770, 1774) to the current image data. Accordingly, theoptimization process 1780 includes one or more feedback including amulti-scale feedback 1784, a synthetic AP slot image feedback 1788, anda synthetic lateral slot image feedback 1792. Thus, the synthetic slotimages 1770, 1774 may be updated to optimize a match to the currentimage data 1744. The multi-scale masking 1760 may be updated, asdiscussed further herein, to optimize the synthetic slot images 1770,1774 in the optimization subroutine 1780 to achieve an optimizationsimilarity to the current image data 1744.

Therefore, the subject registration 1710 may output a transformation ofthe current image data, including the AP slot images 1744 a and thelateral slot images 1744 b, to one another and to the preoperative imagedata 1740 according to the transformation 1796. The transformation 1796may then be output to the device registration process 1720 to registerdevices in the current image data to the preoperative image data 1740 toassist in following a procedure and/or confirming a plan for aprocedure.

As discussed above, the subject registration process 1710 may include asubroutine 1750 to optimize the similarity or generation of syntheticslot images 1770, 1774 relative to the current image data 1744. As apart of the optimization subroutine 1750, the multi-scale masking 1760subprocess is further carried out. In the multi-scale masking 1760 aplurality of masking steps and/or progression of masking steps occurs.With continued reference to FIG. 12 and additional reference to FIG. 13, the multi-scale masking subroutine 1760 will be described in furtherdetail. It is understood that the multi-scale masking 1760 described inFIG. 13 and herein may and/or is incorporated into the optimizationsubroutine 1750, discussed above. Therefore, the multi-scale masking1760 may be understood to be a part of the subject registration 1710.

The multi-scale masking (hereby referred to as masking) may occur in aplurality of stages or steps wherein each stage masks a selected numberof vertebrae for generation of the synthetic slot images 1710, 1774 forthe optimization in block 1780. It is understood that the illustrationin FIG. 13 includes three stages referred to as stage K=1 1820, stageK=2 1824 and stage K=3 1826. Each of the stages 1820, 1824, and 1826 maybe referred to as a selected number of vertebrae that are masked. It isalso understood that the subject registration 1710 may refer toregistration of patient subject images of a spinal column, as discussedherein. In various embodiments, however, the subject registration mayinclude registration of a non-human subject and/or non-spinal elementsin a human or animal subject. Accordingly, the reference herein tovertebrae is merely exemplary. For example, any appropriate identifiedfeature or labeled feature in the images may be registered.

It is also understood that the three stages are also exemplary. More orfewer stages may be used. The selected number may be based upon a speedof computation, achievement of registration convergence time, confidencein registration, or other appropriate factors. For example, a greaternumber of stages may reduce the number of masked portions from stage tostage, while increasing computational time, but may achieve greaterconfidence in registration. Further, fewer stages may decreasecomputational time and increase the number of elements removed per stagebut may have a reduced confidence in registration. It is understood,therefore, that an appropriate number of stages may be selected forvarious purposes.

In general, the multi-stage masking 1760 allows for registration and/orefficient registration between a first image and a second image wherefeatures are not at the same positions relative to one another betweentwo images. For example, a preoperative image data 1740 may be acquiredof the subject 28 at a period of time prior to an operative procedure,which may be proceeded by hours or days. Moreover, a subject may bemoved to a convenient position for an operative procedure that isdifferent than the position for acquiring the preoperative image data1740. Accordingly, the current image data 1744 that may includeintraoperative or post-operative images the image of the subject 28 mayinclude features that are at different relative positions than in thepreoperative images 1740. The masking procedure 1760 allows forachieving a registration between a preoperative image data 1740 and thecurrent image data 1744 when the features are at different relativepositions, such a due to movement of the subject 28.

The registration process 1710 allows for determining a transformation ofthe preoperative image data 1740 such that it matches or is similar tothe intraoperative image data 1744. Accordingly, the transformation mayinclude a mathematical definition of a change or transformation betweenthe two image data and, as discussed further herein, may be directed toa plurality of vertebrae and for a single vertebrae, and sequentiallyfrom plurality to a single vertebrae. Therefore, a single vertebraewithin the preoperative image data 1740 may be registered to a singlevertebrae in the current image data 1744. The single vertebrae isgenerally defined or identified as the same vertebrae in both thepreoperative image data 1740 and the current image data 1744. Theregistration allows the portions identified (e.g., segmented) in thefirst image to be overlayed (e.g., superimposed) on the same portion inthe second image.

The preoperative image data may generally have labeled features thereinthat will be similar or identical to the labeled features in the currentimage data 1744. As discussed above, features may be labeled in theimage data according to the various machine-learning processes. Themachine-learning processes may be used to identify or label the featuresin the preoperative image data 1740 and/or the features in the currentimage data 1744. Therefore, the machine-learning procedures may betrained with preoperative image data or a selected type of preoperativeimage data such as CT, MRI, or the like. For example, the preoperativeimage data may be 3-dimensional image data while the current image datamay be 2-dimensional image data. Further, the features in thepreoperative image data may also be labeled by a user. For example, auser, such as a surgeon or technician, may identify vertebrae, includingvertebral centroids, and label them in a preoperative image data. Thefeatures may also be identified by other appropriate mechanisms oralgorithm such as using a neural network method for automaticallylabeling vertebrae in 3D images. Various techniques may also includethose disclosed in Huang, Y., Uneri, A., Jones, C. K., Zhang, X.,Ketcha, M. D., Aygun, N., Helm, P. A. and Siewerdsen, J. H., “3Dvertebrae labeling in spine CT: An accurate, memory-efficient (Ortho2D)framework,” Phys. Med. Biol. 66(12) (2021), incorporated herein byreference.

As an introduction, the masking process 1760 may end with the finalstage where a single element, such as a vertebra, is a local region ofsupport and may also be referred to as masked. The final stage 1826 maybe a third stage as illustrated above. However, more stages or lessstages may be used. Moreover, the final stage may be achieved after anintermediate stage where only one or two vertebrae are masked relativeto the target vertebra as illustrated in step 1824. This may be precededby a stage where a plurality of vertebrae may be masked. In variousembodiments, in the first stage 1820 an entire range of view or field ofview may be masked as a single element to initiate a rigid registration.It is understood that an identified feature within the full field ofview, such as a labeled vertebra by a user in the 3-dimensional image,may be used to identify a target vertebra. Accordingly, a plurality ofsegments including vertebrae around the target vertebra may be maskedtogether for the masking process 1760.

Further, it is understood that masking an entire field of view may maska plurality of elements that may be later individually masked, such asin the individual mask step 1826. Accordingly, for example, if 15vertebrae are identified the process 1760 may be carried out for each ofthe 15 vertebrae to allow a target (e.g., selected one or more) vertebrato be individually masked in the final stage 1826 for each vertebraidentified in the input image data. Therefore, the procedure 1760illustrated for a single exemplary element, such as a vertebra, ismerely exemplary and may be carried out a number of times necessary foreach element within an image.

The process of the multi-step masking 1760 will be described in greaterdetail with continuing reference to FIG. 13 and additional reference toFIG. 14 . As noted above, the elements in the pre-operative data 1740may be identified, such as with the vertebral centroids in block 1742.Accordingly, the identified features may also be segmented, such as thevertebra may be segmented within the pre-operative image data 1740. Thepre-operative image data may then be rigidly registered to the currentimage data in a selected manner, such as discussed above, as exemplaryillustrated in FIG. 14 in frame 1834. Therein, the pre-operative imagedata may be segmented or otherwise identified, such as identifying edgesor boundaries, and illustrated relative to the current image data 1744.In the rigid registration illustration, the vertebral centroids 1742 maybe identified as elements 1836 relative to the current image data 1744.It is understood, however, that the rigid transformation need not beillustrated and may simply be identified or created for the process 1700and stored internally on a memory to be accessed by the processor.

The rigid transformation may allow for an initial placement of thevertebra or selected elements relative to the current image data 1744.Accordingly, at the first step 1820, five vertebrae may be maskedrelative to a selected vertebra, such as the vertebra L1 1840. Herein,while the vertebra L1 may be the patient vertebra, being registered,alone or with the other vertebra, for the generally discussion thespecific member is identified as “M” and those superior and informationrelative there to as +n and −n, where “n” is the number away from thespecific member M. The masked vertebra or selected vertebra in step 1820may be masked relative to the selected or identified vertebra 1840 andin the appropriate number, such as including two superior and twoinferior relative to the selected vertebra 1840. Accordingly, theselected vertebra elements may be generally referred to as theidentified elements and a selected element plus or minus the identifiedelement. In various embodiments, as illustrated in FIG. 14 , specificvertebrae may be identified. In the current example, if the vertebra L1is identified as the target vertebra 1840, the other four maskedvertebrae may be include the two vertebrae immediately superior of thevertebra L1 (M) which may include two superior vertebra T12 (M+1) 1844and T11 (M+2) 1846 and two inferior vertebra L2 (M−1) 1850 and L3 (M−2)1854. It is understood, however, in various instances a spinal elementmay have been removed or fused and the vertebrae may not be the normalvertebrae. Nevertheless, in various embodiments, the adjacent vertebraemay include two superior and two inferior vertebrae, as noted above. Invarious embodiments, a selected number of vertebrae may include a totalother than five and a different selected number of inferior and superiorvertebrae. Further, as discussed herein, further sub-portions of theSteps K=1 and K=2 include different vertebrae masked relative to thetarget vertebrae.

The masks used in each of the stages 1820, 1824, 1826 of the maskingprocess 1760 may be volumetric masks that are defined relative to thecentroids 1742 in the pre-operative image data 1740. The centroids 1742or appropriate labeled portions can be accomplished via manual methods(e.g., labeling by a surgeon) and/or by automatic methods, includingthose based on appearance models, probabilistic models, andconvolutional neural networks as discussed in Klinder, T., Ostermann,J., Ehm, M., Franz, A., Kneser, R. and Lorenz, C., “Automatedmodel-based vertebra detection, identification, and segmentation in CTimages,” Med. Image Anal. 13(3), 471-482 (2009); Schmidt, S., Kappes,J., Bergtholdt, M., Pekar, V., Dries, S., Bystrov, D. and Schnorr., C.,“Spine Detection and Labeling Using a Parts-Based Graphical Model,”Bienn. Int. Conf. Inf. Process. Med. Imaging, 122-133 (2007); Chen, Y.,Gao, Y., Li, K., Zhao, L. and Zhao, J., “Vertebrae Identification andLocalization Utilizing Fully Convolutional Networks and a Hidden MarkovModel,” IEEE Trans. Med. Imaging 39(2), 387-399 (2020); and/or Huang,Y., Uneri, A., Jones, C. K., Zhang, X., Ketcha, M. D., Aygun, N., Helm,P. A. and Siewerdsen, J. H., “3D vertebrae labeling in spine CT: Anaccurate, memory-efficient (Ortho2D) framework,” Phys. Med. Biol. 66(12)(2021), all incorporated by reference.

In various embodiment, the masks may be defined in an appropriatemanner, and the following are exemplary masks. A process of defining avolumetric mask with a 3-D spline curve fitted to the centroids in thepre-operative image data 1740 may be performed with no additional userinput. Accordingly, the centroids may be defined and the masks may bedefined relative thereto as a 3-D spline curve. A volume of the mask maygenerally be defined as 5 cm×5 cm×3.5 cm that define a volumetric regionabout the fitted curve. In various embodiments, thresholding may also beperformed to remove non-bone tissue, such as defining an intensity tothreshold for the bone. It is understood, however, that otherappropriate thresholds and/or other appropriate volumetric regions or2-D regions may be used to define masks for various types of image data.Further, the various steps 1820, 1824, 1826 may include cropping of thepre-operative image data 1740, the synthetic images 1770, 1774 therefromdue to the masking regions and/or the current image data 1744 tominimize memory usage regarding the target 1840 and the respectivelimited number of masked regions relative thereto.

In the masking procedure 1760, the target vertebrae 1840 masked in thestep 1826 may include a process where an average is identified or usedrelative to a selected number of vertebrae relative to the targetvertebrae 1840 in the prior two steps 1820, 1824. For example, in a mainor primary path 1880 two superior and two inferior vertebrae may beidentified. In a first auxiliary path 1884 one inferior and threesuperior vertebrae may be identified, including a further superiorvertebra 1888. In a further auxiliary path 1892 a single selectedsuperior vertebra 1884 may be identified and three inferior may beidentified including a third inferior vertebrae 1896. Therefore, theprimary and the auxiliary paths 1880-1892 may be used to generateinformation regarding a registration of the target vertebrae 1840 andthe final single masking step 1826.

Accordingly, as illustrated in the process 1760, the final registrationof the target vertebrae 1840 may include an average of threetransformations that occur along the respective paths 1880, 1884, and1892. The primary path 1880 initializes with five vertebrae two superiorand two inferior to the target vertebrae 1840. The first and secondauxiliary paths 1884, 1892 register the target vertebrae 1840 withdifferent or including different vertebrae to register the targetvertebrae 1840 to the current image data 1744. Therefore, after theinitial step 1820 masking, the five vertebrae including the targetvertebrae 1840, three respective transformations are generated toregister the target vertebrae to the current image 1744 and forinitialization of the second step 1824 including three vertebrae. Inthis manner, the primary path 1880 generates a primary transform 1900.The first auxiliary path 1884 generates a second transform 1904 and thesecond auxiliary path 1892 generates a third transform 1906. Therespective transforms 1900-1906 initialize the registration and thesecond step 1824. Therefore, the initial transform 1820, as illustratedin FIG. 14 , includes a registration that may have an error relative tothe current image data for all of the vertebrae but may be minimized forthe target vertebrae 1840. Further, the registration 1820 may be savedin a memory for access for further steps and/or displayed on a displaydevice, as illustrated in FIG. 14 . It is understood that it is notrequired to be displayed for the process 1760.

Following the initial transforms 1900-1906, the second stage k=2 1824may occur with masking of the target vertebrae with only two vertebraerelative thereto. Accordingly, the target vertebrae 1840 is identifiedand masked along with two vertebrae relative thereto. In the primarypath 1880, one inferior and one superior vertebra is masked 1850 and1844. In the first auxiliary path 1884, the two superior vertebrae aremasked 1844 and 1846 in addition to the target vertebrae 1840. In thethird auxiliary pathway, the target vertebrae 1840 is masked with thetwo inferior vertebrae 1850 and 1854. Accordingly, the second stage 1824masks three vertebrae in each of the three paths 1880-1892. Again, eachof these allow for a transformation to register the target vertebrae1840 to the current vertebrae, as illustrated in FIG. 14 at 1824. Eachof the paths generates a respective transform including the primary path1880 generating the transform 1920, the first auxiliary path generatingthe transform 1924 and the third auxiliary path 1892 generating thetransform 1928. Again, each of the transforms 1920-1928 allow for aregistration regarding the target vertebrae 1840 to the current image1744 including information regarding the respective two other vertebrae.

Each of the three transforms 1920-1924 are averaged to a transform 1930.The average transform 1930 is an estimated transform that is computed byaveraging a 3×1 translation vector along each degree of freedom (DOF)and a 3×3 rotation matrix. The average transform is computed using thearithmetic mean of each DOF, and the average rotation is calculatedusing the chordal L₂ mean as disclosed in Hartley, R., Trumpf, J., Dai,Y. and Li, H., “Rotation averaging,” Int. J. Comput. Vis. 103(3),267-305 (2013), incorporated herein by reference. Therefore, the averagetransformation 1930 may be used to initialize the final step 1826 forgeneration of the transformation of the target vertebrae to the currentimage data 1744.

The transformation of the target vertebrae 1840 to the target image datamay be illustrated at 1826 in FIG. 14 and includes a mask that surroundsor includes the single target vertebrae 1840. The single targetvertebrae may be registered to the current image data 1744 by atransformation 1940. The transformation 1940 includes informationregarding the transformation of the single target vertebrae 1840 fromthe pre-operative image data 1740 to the current image data 1744. Asillustrated in FIG. 14 , the process 1760 may be carried out for each ofthe identified vertebrae in the pre-operative image data 1740 to allowfor transformation of each of the individual vertebra to the currentimage data 1744. Thus, the transformation step 1826 may occur for eachof the vertebrae in the field of view of the pre-operative image data1740 to register it to elements in the current image data 1744.

As noted above, the masking process 1760 allows for a transformation ofan individual vertebra even though a deformation (i.e., a change inrelative position of a registered element) has occurred between thepreoperative image data 1740 and a current image data 1744. Asillustrated in FIG. 14 , in the rigid registration 1834, thepre-operative image data may include a registration mismatch relative tothe current image data 1744 as deformation is not accounted. Therefore,the deformation may be accounted for by the multi-mask process 1760.

Moreover, an efficiency may be included by increasing a resolution ofthe respective image data, including the pre-operative image data 1740and the current image data 1744 between each of the sets 1820, 1824,1826. That is the first registration step 1820 include a more coarse orless resolution relative to the final step 1826. This may reducecomputational time and minimize finding of local minima to enhance theregistration of the target vertebrae. Further, it is understood that thetarget vertebrae may be identified in a plurality of the maskingprocesses 1760 for each selected vertebra, which may include all of thevertebrae in the field of the pre-operative image 1740 and/or thecurrent image data 1744.

The registration procedure 1700, as illustrated in FIG. 12 and discussedabove, may register the pre-operative image data 1740 to current imagedata 1744, as also discussed above, and exemplary illustrated in FIG. 14. Further, devices present in the current image data may also beregistered, that is identified in the current image data 1744 andregistered to the pre-operative image data, in part 1720 of theregistration 1700. An exemplary illustrated device may include a medicalscrew 2000, illustrated in the targeted vertebra 1840 in FIG. 14 .Various pre-known or pre-determined information regarding the device2000 may also be used in the registration and proper illustration of apose of the device 2000 relative to the pre-operative image data 1740.This may assist in confirming and/or identifying a procedure relative tothe subject 28.

With continuing reference to FIG. 12 and additional reference to FIG. 15, the device registration to the image data, including the registeredpre-operative image data 1740 may occur with and/or subsequently to theregistration of the pre-operative image data 1740 to the current imagedata 1744. As discussed above, the current image data may be acquiredduring an operative procedure which may include the placement of variousinstruments, such as the medical screw 2000. The device registrationportion 1720 of the registration 1700 may include an input of thecurrent image data 1744 which may include image data of the devices,such as the medical screw 2000 and input of the transform or registeredpre-operative image data, according to the procedure 1710 as discussedabove. The input in the device registration 1720 may include the currentimage data and the registered image data to an optimization procedure2010 which may be similar to the optimization procedure as discussedabove. Generally, the optimization is a gradient correlation (GC) basedupon known parameters also referred to as known components (KC) of thedevice.

The device registration 1720 further includes an input of a device model2020. The device model 2020 may include known components of the device,such as the medical screw 2000. The known components may be based uponthe parameters of the device, such as known dimensions, materials, rangeof relative motion (e.g., a pedal screwhead relative to a shank), etc.In various embodiments, for example, the device 2000 may include thedevice model 2020 that includes 10° of freedom of movement of the pedalhead relative to the change and this may included in the knowncomponents. Others may include six degrees of freedom of position for ascrew shaft, three degrees of freedom of position for rotation of atulip head relative to the screw shaft, and one degrees of freedom ofposition for translational offset between the tulip head and the shaft.Known components may be determined or evaluated according to varioustechniques such as that disclosed in U.S. Pat. No. 11,138,768,incorporated herein by a reference. Further, determination of knowncomponents and various degrees of freedom thereof may also include thatdisclosed in Uneri, A., De Silva, T., Stayman, J. W., Kleinszig, G.,Vogt, S., Khanna, A. J., Gokaslan, Z. L., Wolinsky, J. P. andSiewerdsen, J. H., “Known-component 3D-2D registration for qualityassurance of spine surgery pedicle screw placement,” Phys. Med. Biol.60(20), 8007-8024 (2015), incorporated herein by reference.

The device model 2020 may be used to create or generate syntheticprojections equivalent to the synthetic slot images 1770, 1774.Synthetic images may be synthetic device slot images 2030. The model maybe projected or a projection of the model may be made with projection2034 to generate the synthetic device slot images. The synthetic deviceslot images may also, therefore, be AP and LAT. The synthetic deviceslot images may then be optimized in the optimized process 2010including generation of additional or altered slot images in theiteration process 2038. Accordingly, once the device model isdetermined, which may be input from a memory system, entered by a user,or otherwise accessed by a processor to form a projection to form thesynthetic device slot images 2030 and then optimized through aniterative process of altering the projections to achieve a similarity,such as a gradient correlation, to the devices in the current imagedata. Once the optimization is achieved a transformation 2050 may beoutput to translate or transform the position of the device, such as themedical screw 2000, to the pre-operative image data.

With continuing reference to FIG. 12 and with additional reference toFIG. 15 , an exemplary registration with and without the multi-scaletransformation is illustrated. As illustrated in FIG. 15 , for example,a multi-scale registration 1700 is shown in solid lines and a rigidtransformation is shown in solid lines. Each of the columns illustrate arespective vertebra, such as in L3 vertebra and a L4 vertebra and thetop row illustrates AP images and the bottom row illustrates lateralimages. As illustrated in FIG. 15 , the registered position with themulti-scale transformation according to the registration 1720 discussedabove, differs from that of the rigid process transformation. The studyperformed found that the multi-scale transformation due to the deviceregistration 1720 was more accurate to a confirmed position of theimplanted device than the rigid transformation. As illustrated in the L3AP view, the multi-scale registration illustrates the device 2000 fardeeper into the vertebrae than the rigid transformation device 2000′.Similarly, in the AP view of the L4 vertebra the multi-scaletransformation of the device 2000 is illustrated completely within thevertebra while the rigid transformation of the device 2000′ isillustrated to have pierced the vertebra. Accordingly, the multi-scaleregistration 1720 more accurately illustrates the confirmed anddetermined position of the device 2000 in the subject 28.

The current image data may not precisely illustrate the position of thedevice 2000 in the subject due to various interferences such as metallicinterference, or other interference. Accordingly, the deviceregistration 1720 including known components of the device from thedevice model 2020 assists in determining a registration of the device2000 with a selected accuracy.

The foregoing description of the embodiments has been provided forpurposes of illustration and description. It is not intended to beexhaustive or to limit the invention. Individual elements or features ofa particular embodiment are generally not limited to that particularembodiment, but, where applicable, are interchangeable and can be usedin a selected embodiment, even if not specifically shown or described.The same may also be varied in many ways. Such variations are not to beregarded as a departure from the invention, and all such modificationsare intended to be included within the scope of the invention.

It should be understood that various aspects disclosed herein may becombined in different combinations than the combinations specificallypresented in the description and accompanying drawings. It should alsobe understood that, depending on the example, certain acts or events ofany of the processes or methods described herein may be performed in adifferent sequence, may be added, merged, or left out altogether (e.g.,all described acts or events may not be necessary to carry out thetechniques). In addition, while certain aspects of this disclosure aredescribed as being performed by a single module or unit for purposes ofclarity, it should be understood that the techniques of this disclosuremay be performed by a combination of units or modules associated with,for example, a medical device.

In one or more examples, the described techniques may be implemented inhardware, software, firmware, or any combination thereof. If implementedin software, the functions may be stored as one or more instructions orcode on a computer-readable medium and executed by a hardware-basedprocessing unit. Computer-readable media may include non-transitorycomputer-readable media, which corresponds to a tangible medium such asdata storage media (e.g., RAM, ROM, EEPROM, flash memory, or any othermedium that can be used to store desired program code in the form ofinstructions or data structures and that can be accessed by a computer).

Instructions may be executed by one or more processors or processormodules, such as one or more digital signal processors (DSPs), generalpurpose microprocessors, application specific integrated circuits(ASICs), field programmable logic arrays (FPGAs), or other equivalentintegrated or discrete logic circuitry. Accordingly, the term“processor” as used herein may refer to any of the foregoing structureor any other physical structure suitable for implementation of thedescribed techniques. Also, the techniques could be fully implemented inone or more circuits or logic elements.

What is claimed is:
 1. A method of detecting and classifying a featurein an image based on a plurality of individual image projectionsgenerated with an imaging system, the method comprising: acquiring theplurality of individual image projections as at least a first view ofthe subject and a second view of the subject; determining proposedregions for the feature is in each image projection of the plurality ofimage projections of the acquired plurality of individual imageprojections; combining the proposed regions of the feature in each imageprojection; classifying the feature and determining a position of thefeature in both the first view and the second view; and outputting thedetermination of the position of the feature in each image projection;wherein each image projection is of the subject output determination isoperable to illustrate the determination of the feature and position ofthe feature.
 2. The method of claim 1, wherein the determination processis a machine learning algorithm operable to recognize the feature in theinput image.
 3. The method of claim 1, wherein the first view isdifferent than the second view.
 4. The method of claim 3, wherein thefirst view is a lateral view and the second view is ananterior-to-posterior view.
 5. The method of claim 3, wherein the firstview is an anterior-to-posterior view and the second view is differentthan the anterior-to-posterior view.
 6. The method of claim 3, whereinthe feature is extracted in each of the first view and the second view.7. The method of claim 6, further comprising: determining regionproposals for the extracted features in each of the first view and thesecond view; and aligning the region proposals between the first viewand the second view.
 8. The method of claim 1, wherein the combining theproposed regions of the feature in each image projection includesconcatenating the aligned region proposals of the first view and thesecond view.
 9. The method of claim 8, further comprising: evaluatingthe concatenated the aligned region proposals with a region proposalconvolutional neural network to determine.
 10. The method of claim 9,further comprising: confirming the classification of the featureincluding at least imposing an order on the classification of aplurality of the determined features.
 11. The method of claim 1, whereinthe plurality of images includes the plurality of images acquired with aslot filter; wherein acquiring the plurality of individual imageprojections as a plurality of input images includes acquiring a set ofindividual slot image projections is acquired of a subject with animaging head at a selected position relative to the subject; wherein afilter including a plurality of slots causes the generation of the setof individual slot image projections simultaneously; wherein eachindividual slot image projection of the set of individual slot imageprojections has a unique perspective of the subject relative to eachother individual slot image projection; wherein the first view and thesecond view are both generated with a selected set of the plurality ofindividual image projections.
 12. A system to detect and classify afeature in an image based on a plurality of individual image projectionsgenerated with an imaging system, the method comprising: a processormodule configured to execute instructions to: acquire the plurality ofindividual image projections as at least a first view of the subject anda second view of the subject; determine proposed regions for the featureis in each image projection of the plurality of image projections of theacquired plurality of individual image projections; combine the proposedregions of the feature in each image projection; classify the featureand determining a position of the feature in both the first view and thesecond view; and output the determination of the position of the featurein each image projection; wherein each image projection is of thesubject output determination is operable to illustrate the determinationof the feature and position of the feature.
 13. The system of claim 12,wherein the instructions executed by the processor module to determinethe proposed regions is a machine learning algorithm operable torecognize the feature in the input image.
 14. The system of claim 12,wherein the acquired first view is different than the acquired secondview.
 15. The system of claim 14, wherein the acquired first view is alateral view and the acquired second view is an anterior-to-posteriorview.
 16. The system of claim 13, wherein the feature is extracted ineach of the acquired first view and the acquired second view.
 17. Thesystem of claim 16, wherein the processor module is further configuredto execute instructions to: determine region proposals for the extractedfeatures in each of the first view and the second view; and align theregion proposals between the first view and the second view.
 18. Thesystem of claim 12, wherein the processor module is further configuredto execute instructions to: combine the proposed regions of the featurein each image projection includes concatenating the aligned regionproposals of the first view and the second view.
 19. The system of claim18, wherein the processor module is further configured to executeinstructions to: evaluate the concatenated the aligned region proposalswith a region proposal convolutional neural network to determine. 20.The system of claim 19, wherein the processor module is furtherconfigured to execute instructions to: confirm the classification of thefeature including at least imposing an order on the classification of aplurality of the determined features.
 21. The system of claim 12,wherein the processor module is further configured to executeinstructions to: acquire the plurality of images to include theplurality of images acquired with a slot filter; wherein acquiring theplurality of individual image projections as a plurality of input imagesincludes acquiring a set of individual slot image projections isacquired of a subject with an imaging head at a selected positionrelative to the subject; wherein a filter including a plurality of slotscauses the generation of the set of individual slot image projectionssimultaneously; wherein each individual slot image projection of the setof individual slot image projections has a unique perspective of thesubject relative to each other individual slot image projection; whereinthe first view and the second view are both generated with a selectedset of the plurality of individual image projections.
 22. The system ofclaim 12, further comprising: a x-ray source configured to emit a beamof x-rays; a slot filter to split the beam of x-rays into at least twoslot beam of x-rays; and a detector to detect the slot beam of x-rays.